Using Ccache for Fun and Profit

At PSPDFKit, we work with a large codebase: more than 600K lines and growing. Of course, we aim to write compact, efficient code — but the PDF specification is large, and there are many edge cases that need special treatment. And with PSPDFKit 5 for iOS, compile times became a real issue: The addition of a complete PDF renderer slowed them down significantly.

Our Android SDK had the same problem, and I learned about the Ccache project a few months ago when our Android lead introduced it into our stack to fight the ever-increasing compile times of our C++ NDK code.

So in this blog post, I’ll talk about what Ccache is, how to install it, and some advantages and disadvantages of it. However, one more thing before we dive in: If you’re working on large builds, once you’ve gone through this article and set up ccache, you might find our post on distributed builds using distcc to be beneficial.

See also: Faster Compilation with Ccache 4.0

What Is Ccache?

Ccache is a compiler cache that transparently sits on top of the compiler and first checks its cache before falling back on actually compiling. It has a direct mode and a preprocessor mode. There are a few gotchas, since Clang support was only really added in version 3.2, and the current version is 3.2.3. It’s a project with a long history, and its main focus is to be correct before being fast.

A web search for “ccache xcode” mostly yielded outdated information, and after a quick try, I couldn’t get the caching to work, so I gave up. However, as our codebase grew even more complex, test times went from almost unbearable to really unbearable — even though our Jenkins worker cluster now spans almost 10 Macs. After ranting on Twitter that administrating Jenkins is now basically a full-time job, Christian Legnitto from Facebook (who used to manage Apple’s OS X releases) suggested I give Ccache another go. So here we are.

Let’s Get Started

Installing Ccache is simple: brew install ccache does the job (assuming Homebrew is already installed).

To make Xcode aware of Ccache, we’ll use a small script that configures some environment variables and then invokes Ccache. Save this somewhere in your project and name it ccache-clang:

#!/bin/sh
if type -p ccache >/dev/null 2>&1; then
  export CCACHE_MAXSIZE=10G
  export CCACHE_CPP2=true
  export CCACHE_HARDLINK=true
  export CCACHE_SLOPPINESS=file_macro,time_macros,include_file_mtime,include_file_ctime,file_stat_matches
  exec ccache /usr/bin/clang "$@"
else
  exec clang "$@"
fi

Depending on your setup, you’ll also need a variant named ccache-clang++, in which you replace clang with clang++ for C++.

While this looks a bit complex, it does the right thing and falls back on the regular compiler if Ccache isn’t installed, so new developers can build the project instead of being greeted with “ccache not found” errors. (type is a built-in shell, so the check is fast.)

Be sure to study the configuration page, as there are many options to try. We’re using aggressive caching and it’s working well. For your own project, you might start out without CCACHE_SLOPPINESS and add that once everything is working.

The most important parameter here is CCACHE_CPP2 — this works around a problem where Clang would process preprocessor-processed file outputs and potentially find many code issues that you’d otherwise never see, like unneeded parentheses due to macro expansion. Using this option slows down compile times slightly, but they’ll still be much faster than not using Ccache at all. Peter Eisentraut has an excellent writeup about this.

You also need to define the CC variable in Xcode. At PSPDFKit, we do this in our .xcconfig files, which are shared across all our projects. (This is great to have a unified project configuration and it’s something that’s easily readable and diffable.) However, you can set this inside your Xcode project settings instead:

CC = "$(SRCROOT)/../Resources/ccache-clang";

That’s all! The next full rebuild will be a bit slower, and you can run ccache -s to see if things are actually working. Initially, there should be a lot of cache misses, but when the cache starts to fill up, subsequent compilations will run a lot faster.

Caveats

It’s not all golden: Ccache comes with a few downsides. There’s no support for Clang modules, and Ccache bails if it detects the -fmodules flag at all. You’ll need to go through your code and replace those pretty @import UIKit; lines with the old but compatible #import <UIKit/UIKit.h> — including all the downsides (such as macro overrides) that this brings.

However, at PSPDFKit, we use a lot of Objective-C++, and modules are mostly broken when using C++, so this didn’t affect us. After turning off modules, you may need to add framework references that Xcode picked up automatically before. This is annoying, but it’s also done pretty quickly.

Additionally, you need to stop using precompiled headers. These are no longer recommended by Apple, and they’re generally considered bad style: It’s better be smart about what to import and where. It was rather easy for us to remove our few remaining uses of precompiled headers.

And of course, Ccache doesn’t help you at all for Swift files. While the system that compiles Swift files also uses Clang, it’s a different fork, and Ccache has no idea about Swift files. Maybe this will change one day, but I wouldn’t count on it. Since Swift is still such a fast-moving target and not even binary compatible between minor releases, we can’t use it to build our SDK, so this wasn’t an issue for us either.

It’s always a good idea to check the Ccache status during compile runs to see if any project emits incompatible flags. See the “unsupported compiler option” option. It took me quite a while to clean up all our projects. Setting the CCACHE_LOGFILE environment variable temporarily can be extremely helpful to see exactly what is wrong; Ccache will tell you which flags it doesn’t like and when cache hits and misses are occurring:

steipete@steipete-rmbp ~ $ ccache -s
cache directory                     /Users/steipete/.ccache
primary config                      /Users/steipete/.ccache/ccache.conf
secondary config      (readonly)    /usr/local/Cellar/ccache/3.2.3/etc/ccache.conf
cache hit (direct)                 42530
cache hit (preprocessed)           18147
cache miss                         28379
called for link                     1344
called for preprocessing             645
compile failed                         1
preprocessor error                     2
can't use precompiled header        2567
unsupported source language           12
unsupported compiler option        11564
no input file                          2
files in cache                    124223
cache size                           8.7 GB
max cache size                      15.0 GB

Is It Worth It?

To give you an idea of whether or not this is all worth it: With Ccache, our Jenkins test workers run in an average of 8 minutes, while they needed about 14 minutes before. Compiling and packaging PSPDFKit in all its variants used to take 50 minutes on the fastest MacBook Pro money can buy, but with Ccache, this went down to 15 minutes. Adding Ccache to our stack has been a huge win, and I’m amazed that I hadn’t heard about it earlier. What a great tool!

Precompiled Header Issues

Anton Bukov wrote to say that he resolved some issues by disabling GCC_PRECOMPILE_PREFIX_HEADER but keeping GCC_PREFIX_HEADER.