A real-world WebAssembly benchmark

Nutrient’s mission is to provide the best way to view, annotate, and fill out forms in PDF documents on any platform. In late 2016, we released the first version of our Web SDK, which relies on a server component to render documents. Only a few months later, we released an updated version of Nutrient Web SDK — one that doesn’t require a server component and instead uses WebAssembly (Wasm, WA) to render documents directly in the browser. This was a significant achievement for us, lowering the barrier to entry for our customers by offering a fully browser-based solution.

Rendering PDF documents is a complex task due to the format’s 25+ years of evolution and many edge cases. Rewriting 500,000 lines of C++ code in JavaScript would be impractical, but WebAssembly enables us to use the same codebase in a browser while maintaining high-fidelity rendering and parsing of PDF documents.

Since day one, WebAssembly performance has been crucial to us, and it’s been incredible to see the entire industry focus on optimizing it — from improving WebAssembly startup time, to compiler enhancements and ongoing browser updates.

A real-world WebAssembly benchmark

As part of our commitment to making WebAssembly faster, we developed a WebAssembly benchmark to provide browser vendors with a real-world, open source test based on Nutrient Web SDK.

The benchmark is available on GitHub and works with both customer and trial licenses. Browser vendors can reach out to us and obtain a more permissive license key so that the benchmark can run on different machines and even on their continuous integration servers.

The benchmark is a simple JavaScript application that runs in the browser. Using the Web Performance API, we measure the time of various standard actions on a PDF document. The results are categorized into loading time (compilation and instantiation) and runtime, with compilation times being the primary differentiator among browsers.

Considerations

Before we dive right into analyzing our results, we want to share more details about our test setup. One thing we’re trying to accomplish with this benchmark is to have a score we can use to improve the performance of Nutrient Web SDK instead of creating a microbenchmark. As such, the benchmark does not call directly into WebAssembly; rather, it also tests the bridge we use to communicate with it — an important part of Nutrient Web SDK.

Additionally, we had to disable one of the major improvements to WebAssembly startup time: streaming compilation. This makes it harder to compare between the WebAssembly results and our JavaScript-fallback¹ results, as the latter can’t use this optimization. We want a score that can easily be compared across a wide variety of browsers, and to make this possible, we had to remove the network speed noise from the equation.

Our WebAssembly payload is also rather unusual. Much of the size of our WebAssembly artifact is attributed to special cases in the PDF spec that might not be executed for every document. We want our PDF viewer to deliver the best results no matter the device. This is a factor where asm.js excels, as it only needs to compile functions when they’re actually run.

Results

The results were a bit surprising at first. While WebAssembly is making great progress, when we compare WebAssembly with our JavaScript fallback for the same product and tasks, all browsers except Firefox are still slower.

After collecting the first results from our benchmarks, we reached out to the individual browser vendors to discuss these results and investigate ways to further improve the numbers. At this point, we want to thank Google, Mozilla, Microsoft, and Apple — all browser vendors were exceptionally helpful along the way and which provided valuable feedback to help improve our benchmark.

Chrome

Runtime performance of WebAssembly has always been great when using Google Chrome. We were only a bit disappointed to find out that the v8 team abandoned the plan to implement IndexedDB caching for WebAssembly.

However, after Google invited us to a call and shared some of the company’s upcoming plans, we’re excited about what’s in store. The biggest change in the near future will be the introduction of a new baseline WebAssembly compiler. You can already try this compiler today by running the latest canary build and enabling the enable-webassembly-baseline flag. In our internal tests, we’ve noticed significant improvements in the total startup time. We couldn’t be happier to see this compiler being enabled by default, starting with the next canary version (M69).

Additionally, the team gave us a sneak peek at the upcoming alternatives for caching the compiled WebAssembly module so that you don’t have to recompile on every browser refresh.

Update August 2018 — Google released Liftoff, a new baseline compiler for WebAssembly in V8. The announcement blog post includes specific benchmarks for Nutrient, showing a 56%+ faster initialization time.

Firefox

Firefox delivered the best results in our benchmark, no doubt due to the release of tiered compilation, along with IndexedDB caching.

The team was especially helpful in pointing out bottlenecks in our JavaScript implementation, and we’re already incorporating these changes into our next Nutrient Web SDK release.

The team also pointed out that after a very fast initial compilation, browsers that use a baseline compiler kick off a slower, more optimizing compilation in the background that continues to execute while we run other benchmarks. For this reason, we now benchmark initialization at the end to reduce the amount of background interference.

Update October 2018 — Starting with Firefox 63, the team at Mozilla has added several optimizations that improve the performance of calls between JavaScript and WebAssembly. The Nutrient score for this version was not updated, as we couldn’t measure noticeable performance differences; we’re simply not calling between JavaScript and WASM often enough.

Edge

The performance of Edge was underwhelming at first. But with recent optimizations like inlining for WebAssembly, Microsoft Edge proves that the company is fully committed to making WebAssembly faster in the future.

Microsoft told us that the company is currently working on making WebAssembly easier to produce, consume, and extend. In the future, Microsoft wants to bring more features — which are currently not possible in asm.js (or JavaScript) — to WebAssembly.

In addition to Edge, we made sure our JavaScript fallback also runs (just like our SDK) on Internet Explorer 11. If you want to try it out, make sure to have a coffee ready, as the performance will be quite a bit slower (in our tests, it was 6.5x slower).

Internet Explorer 11 is no longer supported in our Web SDK as of version 2022.5. Edge 18 is no longer supported in our Web SDK as of version 2023.2.

Update August 2018 — The WebAssembly team at Microsoft worked with us to include optimizations for our WebAssembly benchmark in Edge 44, which is now accessible via the Windows Insider build. This results in both an impressive 2.25x faster benchmark run over the previous version, and a Nutrient score that is now on par with Google Chrome’s new baseline compiler. We’ve updated our bar chart to include these numbers.

Safari

We’ve found that Apple’s Safari’s performance is especially bad on beta versions of macOS, and we worked with Apple’s engineers to track down a recent regression (Bug 187196) where trace points turned out to be the expensive factor. Apple has also been exceptionally awesome in reacting to this bug, and the fix has already landed in master.

Conclusion

With this blog post, we want to say thank you ❤️ to all the browser vendors for their efforts to make WebAssembly fast and successful. We also want to communicate that our door is always open and we are here to collaborate with you to make our product and the web platform better!

If you like this article, be sure to also check out our other related blog post: WebAssembly: A New Hope and Optimizing WebAssembly Startup Time.

Regardless of whether or not you’re a browser vendor, try out the WebAssembly Benchmark and feel free to reach out in case you have any feedback.

¹ JavaScript fallback refers to our asm.js build with ALLOW_MEMORY_GROWTH=1. Because of the dynamic nature of PDFs, some of them might need a lot more memory than others. Our internal tests have shown that this option works best in our use case.

FAQ

Here are a few frequently asked questions about Nutrient’s WebAssembly benchmark.

What is the purpose of the WebAssembly benchmark in this post?

The WebAssembly benchmark aims to help browser vendors assess WebAssembly performance through real-world tasks.

Why did Nutrient choose WebAssembly for rendering PDFs?

WebAssembly allows high-fidelity PDF rendering by reusing Nutrient’s C++ codebase directly in the browser, enhancing speed and accuracy.

Does this benchmark support all major browsers?

The benchmark is compatible with most modern browsers, and Nutrient collaborates with browser vendors to improve support.

What insights were gained from benchmarking across browsers?

The benchmark revealed differences in WebAssembly loading and runtime performance, which vary by browser due to each browser’s unique handling of WebAssembly.

How can browser vendors use this benchmark?

Browser vendors can reach out to Nutrient for a permissive license to run the benchmark on various platforms and improve WebAssembly performance.