Server Development on Apple Silicon

In my last post, I outlined our journey of getting our server products, PSPDFKit Web Server-Backed and PSPDFKit Processor, running on ARM-based hardware. Today, I want to share how we got Apple silicon working on our Server-based products and give you my perspective of where it works and where it doesn’t.

Getting Set Up

The first thing that needs to be discussed is getting set up. The tech stack we use for our server-based products consists of:

• Erlang and Elixir for running our main Server application

• Our pspdfkitd binary, which is based on our shared core that we use for all PDF-related operations

• Node.js for running tests against our HTTP API

• Ruby for internal tooling around development

Let’s see if we can make this work on Apple silicon.

External Dependencies

We use the asdf version manager for managing all our dependencies. For context, this was our .tool-versions file when we started this:

nodejs 10.15.1
elixir 1.11.2-otp-23
erlang 23.2.1
ruby 2.7.1

Let’s try installing the above on an M1 MacBook — after adding the relevant plugins, of course:

$ asdf install
/Users/user/.asdf/plugins/nodejs/bin/../lib/utils.sh: line 62: printf: write error: Broken pipe
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:-- 100  4229    0  4229    0     0  13511      0 --:--:-- --:--:-- --:--:-- 13511
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:-- 100 10860    0 10860    0     0  32612      0 --:--:-- --:--:-- --:--:-- 32612
Binary not found for version 10.15.1

So we already hit our first road block: The Node.js plugin for asdf relies on prebuilt binaries for each supported platform. Sadly, at the time of writing, there wasn’t a binary available for ARM-based MacBooks. Luckily, Node itself does support ARM, so if you build it yourself, it should work. Let’s try using nvm:

$ nvm install 10.15.1
…
#error Target architecture arm64 is only supported on arm64 and x64 host
 ^
1 error generated.
1 error generated.
make[1]: *** [/Users/user/.nvm/.cache/src/node-v10.15.1/files/out/Release/obj.host/v8_libbase/deps/v8/src/base/file-utils.o] Error 1
make[1]: *** [/Users/user/.nvm/.cache/src/node-v10.15.1/files/out/Release/obj.host/v8_libbase/deps/v8/src/base/functional.o] Error 1
1 error generated.
make[1]: *** [/Users/user/.nvm/.cache/src/node-v10.15.1/files/out/Release/obj.host/v8_libbase/deps/v8/src/base/ieee754.o] Error 1
make: *** [node] Error 2
nvm: install v10.15.1 failed!

Still no luck. After some additional research, I found a suggestion that said using a newer version of Node might actually work. Let’s give that a shot:

$ nvm install 15.11.0
…
Now using node v15.11.0 (npm v7.6.0)
$ node --version
v15.11.0

It took a couple of minutes, but we now have Node running on our M1 MacBook. Let’s remove the asdf plugin and try installing our dependencies again:

$ asdf plugin remove nodejs
$ asdf install
Downloading openssl-1.1.1i.tar.gz...
-> https://dqw8nmjcqpjn7.cloudfront.net/e8be6a35fe41d10603c3cc635e93289ed00bf34b79671a3a4de64fcee00d5242
Installing openssl-1.1.1i...
Installed openssl-1.1.1i to /Users/user/.asdf/installs/ruby/2.7.1

Downloading ruby-2.7.1.tar.bz2...
-> https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.1.tar.bz2
Installing ruby-2.7.1...
ruby-build: using readline from homebrew

BUILD FAILED (macOS 11.2.1 using ruby-build 20201225)

Inspect or clean up the working tree at /var/folders/m3/bbbz7q152k75f4d51lnh5vc00000gn/T/ruby-build.20210311160726.52882.JhuVIJ
Results logged to /var/folders/m3/bbbz7q152k75f4d51lnh5vc00000gn/T/ruby-build.20210311160726.52882.log

Last 10 log lines:
compiling psych_parser.c
compiling psych_to_ruby.c
compiling psych_yaml_tree.c
compiling ../.././ext/psych/yaml/reader.c
compiling ../.././ext/psych/yaml/scanner.c
compiling ../.././ext/psych/yaml/writer.c
linking shared-object stringio.bundle
linking shared-object zlib.bundle
linking shared-object psych.bundle
make: *** [build-ext] Error 2

It’s clear that we’re still in uncharted territory here. After some more research, it turned out that a simple version bump is all that’s required. Changing our Ruby version from 2.7.1 to 2.7.2 seems to do the trick:

$ asdf install
Downloading openssl-1.1.1i.tar.gz...
-> https://dqw8nmjcqpjn7.cloudfront.net/e8be6a35fe41d10603c3cc635e93289ed00bf34b79671a3a4de64fcee00d5242
Installing openssl-1.1.1i...
Installed openssl-1.1.1i to /Users/user/.asdf/installs/ruby/2.7.2

Downloading ruby-2.7.2.tar.bz2...
-> https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.bz2
Installing ruby-2.7.2...
ruby-build: using readline from homebrew
Installed ruby-2.7.2 to /Users/user/.asdf/installs/ruby/2.7.2

$ ruby --version
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [arm64-darwin20]

Another 10 minutes later and we have Ruby running as well. Interestingly, if you check the changelog, you’ll find no references to any fixes made for ARM support, and indeed, some other people report that even 2.7.2 only installs in Rosetta. However, I had no issues, and as you can see from the output, it is indeed running as a native ARM64 binary. Let’s see if Erlang and Elixir will cause issues as well:

$ asdf_23.2.1 is not a kerl-managed Erlang/OTP installation
No build named asdf_23.2.1
Extracting source code
Building Erlang/OTP 23.2.1 (asdf_23.2.1), please wait...
…

Erlang/OTP 23.2.1 (asdf_23.2.1) has been successfully built
Installing Erlang/OTP 23.2.1 (asdf_23.2.1) in /Users/user/.asdf/installs/erlang/23.2.1...

$ erl
Erlang/OTP 23 [erts-11.1.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1]

Eshell V11.1.5  (abort with ^G)
1>

That was painless. Seems like Erlang already supports Apple silicon in our current version. What does Elixir say?

$ asdf install
==> Checking whether specified Elixir release exists...
==> Downloading 1.11.2-otp-23 to /var/folders/m3/bbbz7q152k75f4d51lnh5vc00000gn/T//elixir-precompiled-1.11.2-otp-23.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:-- 100 5720k  100 5720k    0     0  8897k      0 --:--:-- --:--:-- --:--:-- 8883k
==> Copying release into place

$ elixir --version
Erlang/OTP 23 [erts-11.1.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1]

Elixir 1.11.2 (compiled with Erlang/OTP 23)

Now we’re all done, and we have our dependencies installed and running!

The takeaway here is that unless your tech stack is really esoteric (or using very old versions), chances are it should be relatively easy for you to make it work on Apple silicon. If you already try to stay up to date with your versions, you’ll most likely not even need to do anything beyond installing them as usual.

Our pspdfkitd Binary

With all our external dependencies installed, there was one more component we required when working on PSPDFKit Web Server-Backed or Processor, and that’s our PDF processing tool. This is what handles all PDF-related operations. In our release, it’s packaged inside the Docker image we ship, but when working locally, we need it in our path. Usually, we put it in the /usr/local/ folder, but on the M1 MacBook, by default, /usr/local/bin/, /usr/local/share/, and /usr/local/lib/ are owned by root. And since Homebrew also puts ARM binaries in its own /opt/homebrew/ folder, we decided it’s best to follow suit and not require sudo.

We don’t have a native ARM version of pspdfkitd for macOS yet (we do have one for Linux, which we use for our Web Server-Backed and Processor releases), but it runs with no issues in Rosetta 2. So all we had to do was adjust our installation script to use the new paths as outlined above. It looks something like this in action:

if [[ $OSTYPE == darwin* && `uname -m` == "arm64" ]]; then
  # M1 MacBook use /opt/homebrew
  mv /tmp/pspdfkitd /opt/homebrew/bin
else
  # Use /usr/local
  mv /tmp/pspdfkitd /usr/local/bin
fi

We rely on Homebrew having already added /opt/homebrew/ to the path on our development machines, so we don’t need any additional setup.

The Development Experience

Considering setting up wasn’t much harder than on a regular Mac, how is the development experience? It has its ups and downs; let’s take a closer look.

The Good: Writing and Running Code

This is where the new M1 MacBooks really shine: They’re completely silent, they barely get warm, and they’re more than fast enough to handle working on our Elixir codebase. You’ve got all the tools you need to work, since things like Visual Studio Code already have Apple silicon-compatible versions. And even things that don’t yet, such as SmartGit (the git program of choice for me), work perfectly fine in Rosetta 2.

Now, there are already many websites that have benchmarked the performance of the new Apple silicon Macs against their Intel counterparts, so I won’t get too much into that. But if you work with Elixir and are worried about compile times, fear not: I’ve got the data for you. I’m comparing my 2019 16” MacBook Pro (2,4 GHz 8-Core Intel Core i9, 32 GB 2667 MHz DDR4) with my 2020 MacBook Air (M1, 16 GB). Let’s look at the time output of a clean compile of PSPDFKit Web Server-Backed.

2019 MacBook Pro Results:

$ time mix compile
mix compile  299.75s user 21.70s system 362% cpu 1:28.72 total
mix compile  297.25s user 21.79s system 365% cpu 1:27.23 total
mix compile  299.36s user 22.15s system 359% cpu 1:29.33 total
mix compile  310.94s user 22.60s system 362% cpu 1:32.00 total

2020 MacBook Air Results:

$ time mix compile
mix compile  102.73s user 23.90s system 213% cpu 59.451 total
mix compile  103.15s user 22.52s system 199% cpu 1:02.96 total
mix compile  103.15s user 22.52s system 199% cpu 1:02.96 total
mix compile  101.87s user 23.41s system 214% cpu 58.460 total

Now those results are just impressive, though it’s also a slightly unfair comparison. This is because the 2019 MacBook I used for comparison has been my main workhorse for almost a year at this point, and it has much more stuff installed on it and running in the background. Meanwhile, the M1 MacBook Air is essentially straight out of the box with only the bare minimum installed and running. That being said, the giant 16” MacBook taking three times as long for a clean compile is still a bad result, even considering other stuff running.

There isn’t much else to add here. If you work with Elixir and run it straight from the command line (as opposed to in Docker, which we’ll get to next), the new M1 MacBook is great.

The Bad: Docker

This brings me to what isn’t great, and that’s Docker. While there’s a Docker version that runs on Apple silicon, and all images that support ARM64 work with it, it’s much slower than on Intel. For us, during development, that doesn’t really matter, since we run our entire server directly on the Mac. However, on occasion, we have to check if changes we make to our Docker build work, and that means building Docker images.

Let’s compare how long it takes to build our base image using the same hardware as above. Building this Docker image includes compiling all runtime dependencies we need for PSPDFKit Web Server-Backed — such as Ruby — and pulling dependencies via APT.

2019 MacBook Pro Results:

$ time docker-compose build --no-cache base

real    6m11.356s
user    0m0.966s
sys     0m0.366s

real    6m2.826s
user    0m1.004s
sys     0m0.387s

2020 MacBook Air Results:

$ time docker-compose build --no-cache base

real    11m58.859s
user    0m4.526s
sys     0m1.748s

real    12m19.955s
user    0m4.544s
sys     0m1.833s

As you can see, neither is really fast (to be fair, our CI also takes six minutes to build this image), but the M1 MacBook is much slower here, taking twice as long. This, I think, comes down to the file system performance for Docker on Mac being not great, which makes it an even bigger bottleneck on the M1 MacBook. That being said, it’s still early for Docker on Apple silicon, and there isn’t even a stable release out yet, so this surely will get better in the future. For now, I’d recommend using remote builders when using Docker on Apple silicon. As long as your connection is relatively fast, this will be much quicker than trying to build locally, and it’s a great way to see if your build context is bloated.

As for running Docker images, as long as they’re built for ARM64, this works just fine (this includes PSPDFKit Web Server-Backed and Processor as well). There isn’t anything to complain about performance-wise either; it’s just building them that’s too slow for productive work.

With that, let’s wrap up.

Conclusion

Assuming your tools are supported on Apple silicon and you don’t rely heavily on Docker, the new M1 Macs are great for development. Many of the tools we as developers rely on are natively supported, and for those tools that aren’t yet, Rosetta 2 works well. I only looked into Elixir development today, but Apple silicon should do great with almost anything.

I hope this gave you an idea of what to expect if you decide to pick up one of the new M1 Macs for development. Be sure to let us know if there’s anything else we can answer.

If you want to learn more about ARM at PSPDFKit, you can also check out our previous post, where we looked at how we added ARM support to our PSPDFKit Web Server-Backed and Processor products.