Continuous Integration: From Jenkins to Buildkite

We started using Jenkins for CI/CD at PSPDFKit in 2013, and it served us well for many years. As we grew and released more products, we eventually ran into limitations and ultimately moved to Buildkite in 2019. We’ve been using Buildkite for almost two years, so it’s time to reflect on our choice and take a fresh look at the market.

This article is part of a series about Continuous Integration for Small iOS/macOS Teams.

In the Beginning: Jenkins

Jenkins is the most popular continuous integration (CI) system. It’s open source, it’s written in Java, and it can run on any platform. It also has a market share of almost 60 percent. Jenkins served us well for many years, but it requires a significant amount of maintenance, and it has various architectural shortcomings that make it difficult to work with.

Our main issues were around updating plugins and Jenkins itself. Without a staging Jenkins instance, it’s really easy to crash the complete instance when updating one faulty plugin. As a result, the appeal to try and improve our Jenkins CI with available plugins was low, and along with this came a general high risk that we’d break our production CI. Installing a plugin also requires a restart of Jenkins, which meant that when doing this, all ongoing builds needed to be stopped.

A Jenkins instance consists of a large number of plugins that are all developed and updated independently of each other. This leads to available plugin updates every few days. Despite those updates, we got the impression that Jenkins’ ecosystem is slowly dying: A lot of the plugins we relied on are either barely maintained or not maintained at all.

The Jenkins Web UI is mostly configured via complex web forms, and editing changes in bulk was difficult, as not every setup is supported via the newer Jenkinsfile concept.

At the time of evaluation, there was also uncertainty about the possibility of a larger rewrite, which would mean additional potential breaking changes.

CI Solution Requirements

To evaluate what else we could use, we wrote up our requirements for a Jenkins successor:

• Support for agents on all major platforms — Docker, Linux, Windows, macOS

• Pipeline support (parallel steps, artifact support)

• First-class container/Docker support

• Integration with GitHub PRs

• Easy to use

• Flexibility/customizability

• Build cache support (ccache)

• Pricing per user, not per agent

What follows is a list of possible solutions we looked into, along with their pros and cons.

Bitrise

Bitrise advertises heavily in the mobile space and focuses on mobile deployment. It’s a full-featured solution that comes with its own macOS workers.

Hardware is either 2vCPU @ 2.7 GHz, 4 GB RAM (Standard) or 4vCPU @ 3.5 GHz, 8 GB RAM (Elite). Bitrise charges $3,240/year for two concurrent builds, and each extra concurrent build is $1,200/year.

We stopped the evaluation there as the hardware isn’t powerful enough for our needs, and there’s no support for Windows.

CircleCI

The fastest macOS hardware for CircleCI is 4vCPU, 8 GB RAM or 8vCPU, 16 GB RAM. But it comes with a caveat: “This resource requires review by our support team.”

At the time of our initial evaluation, there wasn’t yet support for Windows. However, it was added in late 2019.

CircleCI doesn’t support custom agents, and the macOS hardware isn’t fast enough, so we didn’t evaluate it more.

TeamCity

TeamCity by JetBrains is an extremely powerful CI system. The Professional version is free, but to meet our requirements, we’d need TeamCity Enterprise, which costs €21,999 initially, with renewals costing half of the initial price.

The price is high, and the feedback from our team was that it “looks almost as bad as Jenkins,” so we decided to not investigate this option.

GitHub Actions

GitHub Actions is one of the newest CI systems, and it works beautifully with GitHub, as it’s a feature of GitHub. It makes perfect sense that GitHub is investing in the CI arena.

The macOS workers offered are quite slow (2-vCPU, 7 GB RAM, 14 GB disk), but this solution both offers hosted machines and supports self-hosted runners.

If we were just starting CI in 2021, we’d first look into GitHub Actions before investigating another system.

GitLab CI

GitLab CI has lots of features, and it’s possible to create complex pipelines out of the box. It has a head start of a few years over GitHub Actions, and the product feels pretty mature.

The agent is a single Go binary without dependencies. It supports monorepos with only:changed in the pipeline YAML.

GitLab syncs the complete repository to enable its CI solution, which means it would get all our source code unless we were to host GitLab ourselves. This makes sense, as GitLab is a competitor of GitHub, but it’s unattractive when one isn’t interested in moving the version control server. Additionally, clicking on a commit in the CI job will take you to GitLab instead of back to GitHub. This is just one example of how the company tries to get you to change to GitLab completely.

At the time of our initial evaluation, it wasn’t possible to see JUnit results. This is something that has since been implemented.

Pricing is quite reasonable at $19 for the Silver plan, which should be enough for a company of our size. But it’s significantly more than the $4/month we currently pay at GitHub.

Since we have no interest in moving away from GitHub, we ended the evaluation here. If you start a new project in 2021 and are free to choose where you host your source code, this is a really good choice.

Buildkite

Buildkite is an orchestrator, but it’s currently trialing offering macOS workers as well. This is still in early access, so there’s no information available about performance other than an ominous “up to 30% faster than other CI/CD providers.”

The Buildkite system is simple but easy to extend with our own hooks/scripts. Pricing is $15 per user per month ($12.75 if paid annually). It supports GitHub and Google SSO, comes with an extensive API and a GraphQL API, and webhooks are available.

The agent is a single Go binary without dependencies, and it runs on any platform. Buildkite was super quick to offer a version that runs natively on Apple Silicon as well. It’s possible to annotate CI output with whatever information we want, and pipelines can be created dynamically with scripts.

There’s a positive case study by Pinterest for its iOS builds with Buildkite.

One thing we’re missing is an overview of which jobs are running on which machines. It’s possible to collect metrics and create your own dashboards, but there’s nothing out of the box. This was much better with Jenkins, but it’s not something essential. There’s also no way to give specific jobs priority — only agents can be given priority. A suboptimal workaround is to use different queues.

Another downside is Buildkite supports monorepos by using a pipeline that triggers different pipelines depending on the changed files. This is not an elegant solution.

In early 2021, Buildkite started offering managed macOS agents, which are priced at $1,700 per agent per month. This amounts to around $408K/year, which is about 10 times higher than what we currently pay for a similar service.

Conclusion

Two years later, we’re still extremely happy with our choice. Buildkite is fast, stable, and easy to administrate. The worker is straightforward to install and update, and it’s generally much more reliable than Jenkins. But it’s great to see that GitHub is pushing its CI/CD system, and we’ll be reevaluating this again in a few years.

This article is part of a series about Continuous Integration for Small iOS/macOS Teams, where we’re sharing our approach to macOS CI. Make sure to check out the other posts!