Managing macOS Hardware: Virtualization or Bare Metal?

There’s a reason why build minutes are more expensive on continuous integration (CI) services when it comes to macOS: Macs are notoriously difficult to use via CI. Running macOS — even virtualized — requires macOS hardware.

This is not a technical limitation — macOS runs perfectly fine in virtualization on standard PC hardware — but rather a restriction in Apple’s EULA. This restriction, combined with the fact that Apple doesn’t offer server hardware, increases costs and makes running macOS more expensive and challenging.

With the release of Big Sur, Apple changed the license to double down on the macOS hardware requirement. It also officially stipulates that renting software is allowed, just in time for Amazon to enter the space with its new EC2 Mac instances.

This article is part of a series about Continuous Integration for Small iOS/macOS Teams.

Virtualizing macOS

When it comes to macOS virtualization, there are three commercial solutions available on the market:

• VMware vSphere (VMware Hypervisor)

• Veertu Anka (Apple Hypervisor)

• MacStadium Orka (KVM Hypervisor)

It is possible to install macOS with open source solutions such as KVM, but this requires a lot of dedication and — if you factor in time — it’s likely more expensive than commercial solutions.

VMware vSphere is the oldest available solution, whereas Anka (2017) and Orka (2019) are newer technologies. While image-based solutions are intriguing, there’s a performance and complexity overhead when using virtualization, especially with macOS, which wasn’t made to be virtualized. I asked peers on Twitter for their experiences using these technologies. Below I’ll outline some of the considerations.

Performance

Every solution that puts layers between the OS and the hardware will incur a performance overhead. It’s hard to measure exactly how much that is, but reports go from twice as slow to a mere 10 percent overhead. Your VM setup also might be otherwise constrained, resulting in poor networking performance.

With different VMs running on a machine, performance varies based on workloads in other VMs. Spotify moved from VMware to bare metal in late 2019 for performance reasons like total speed and a more predictable environment. Lyft moved to bare metal for similar reasons.

Bugs

With an additional software layer, you’ll inherit bugs. There are reports of memory leaks, “VMs just getting stuck,” and “unreliable USB passthrough.” In cases where a company relies on a system Hypervisor (Anka uses Apple’s Hypervisor framework), there are only limited ways to work around bugs.

Compatibility

Relying on virtualization will also delay adoption of new OS versions or hardware such as Big Sur or Apple Silicon, as the vendor first has to finish updating its product. New features might also cause issues. For example, the iOS Simulator uses Metal for acceleration in Big Sur, but none of the virtualization solutions support a virtualized Metal graphic card. (Veertu is reported to be working on it.)

Nested Virtualization

Not all solutions support nested virtualization, which is needed if you run other CI tests as well. The Android Emulator uses virtualization, and while it seems some services like Anka support this, there’s considerable performance overhead.

Price

I haven’t found a single writeup that takes price into consideration when discussing macOS virtualization. This is in some ways understandable, as most articles are from large companies, and engineers aren’t included in their price decisions. However, for smaller teams without venture capital, it’s an important metric.

Total cost is difficult to measure, since the promise of virtualization is less ongoing work, which should translate to reduced ongoing maintenance costs, and often employees are the most expensive cost factor.

Anka charges around $600 per core per year for its Basic tier (source). There are some resellers that include hardware in their offerings; for two Mac minis, this comes to $17,868 (source).

If you’re installing Anka Build on 20 Mac minis * 6 cores, the yearly license subscription cost will be 20 * 6 * 600 = $72,000/year. The folks at Anka seem really nice, and I’m sure there’s room for negotiation, but it should serve as a reference point that virtualization technology will, in most cases, be more expensive than the actual machines you’re renting.

VMware vSphere also charges per core and seems to be in a similar price range. Orka lists a starting price of $18,000/year on its website and likely scales to a similar range. Since Orka offers Mac hardware and virtualization at once, there might be potential for higher savings.

Comparing Anka and Orka

Anka and Orka are conceptionally quite similar, even though they use different technologies to virtualize macOS. While there’s no neutral comparison website, you can read up on Veertu’s (Anka) take on Orka and MacStadium’s take on Anka. Shopify and Pandora have case studies for Anka, and AppDynamics and Homebrew have case studies for Orka.

One upside to choosing a virtualization provider is that you get great support. I’ve heard good things about both Anka and Orka, with people saying their support is very helpful.

Fully Managed Services

With Big Sur, Apple changed the EULA to define in which ways macOS renting is allowed, and it states a “lease period must be for a minimum period of twenty-four (24) consecutive hours.” This will have implications for future virtualization offerings such as Travis CI, CircleCI, and GitHub Actions, all of which currently have a flexible “build minutes” model. (Meanwhile, GitHub charges 10 times as much for a macOS minute vs. a Linux minute.) It’s also a rather puzzling change, as it requires hosting providers to offer less flexible pricing.

Since fully serviced providers are not as flexible as solutions where you have full control over the OS, we’re not considering them in this comparison.

Bare Metal macOS

As a small team, we didn’t want to deal with the complexities of virtualization, and we need to be able to adopt new operating systems fast to support technologies such as Mac Catalyst, so we chose bare metal. But even large companies — such as LinkedIn, Uber, and Ditto — moved to bare metal.

There’s a variety of providers offering dedicated Mac hardware: MacStadium, MacinCloud, MacWeb, Mac Mini Vault, the Swiss-based Flow (Spotify is a prominent customer), and — the latest addition — Amazon AWS. There are rumors that Microsoft Azure is working on an offering as well.

Pricing

Pricing varies a lot: MacStadium lets you rent the latest, fastest Intel-based Mac mini (6-core 3.2 GHz, 64 GB RAM, 1024 GB SSD) for $159/month or $1,908/year (This is limited time pricing that lasts until 31 December 2020; the minimum commitment is one month).

Amazon currently offers the same model with 32 GB as the only option, with a price of $1.083 per hour (with a minimum commitment of 24 hours as per the Apple EULA) or $9,487 per year. Here’s an interesting detail about Amazon: The company offers image-based restore and connect and supports AWS Nitro via Thunderbolt. With reserved instances, the price can go down.

The Geekbench score of the top-of-the-line Intel Mac mini is 1100/5468 (RAM doesn’t matter here). However, the upcoming Mac mini M1 is even better, with 1702/7380, and it’ll likely be at an even more competitive price point, given that a comparable model sells at $1,299, whereas the Intel configuration retails at $2,499.

For many years, our hosting provider has been MacStadium, and the company is extremely competitive on pricing. A fleet of 20 MacStadium Mac mini machines costs 159 * 12 * 20 = $38,160/year. We currently run 16 of these machines and expect that costs will go down even further once we move to Apple Silicon, as we’ll need fewer nodes to process the same workload, and renting costs should go down as well.

What about the Mac Pro?

You might be asking, what about Mac Pros? We experimented with them, but for the price of one Mac Pro, we get at least six Mac minis, which is far more cost efficient and better for running different tasks. To use a Mac Pro efficiently, you’d need to set up virtual instances or accept that most cores would be idle when running tests. While compiling scales well to multiple cores, linking is mostly constrained to a single core, and even with parallel testing enabled, you can only scale to so many simulators before the system becomes unstable. Spotify experimented with Mac Pro but only got a 30 percent performance increase for 6 times the price.

Automatic Machine Setup with Chef

The biggest downside of running on bare metal is that each machine needs to be set up manually. We mostly automated the process via Chef; other companies use Puppet or Bash or Ruby scripts. What remains are OS-level updates, but this is a small-enough task that it can be done without a dedicated person managing CI.

Learn more about how we use Chef to automate macOS in the next blog post of our CI series: Automating macOS with Chef.

Conclusion

There’s no one-size fits-all solution when it comes to running macOS in the cloud. Both virtualization technology and bare metal are valid choices depending on organizational structure and requirements, but we hope this has given you a good overview of what’s possible.

This article is part of a series about Continuous Integration for Small iOS/macOS Teams, where we’re sharing our approach to macOS CI. Make sure to check out the other posts!