Debugging Image Loading and Cropping Performance on iOS

If you use the PSPDFKit iOS SDK or perhaps the PDF Viewer Pro app, then you may have come across the feature of adding images to a PDF. A user can select an image from the Photos library, crop the image, and then select the resolution of the image to add to the PDF as an image annotation.

Recently, one of our customers contacted us via support to let us know this functionality was sometimes crashing when integrated and used in an app extension. After looking into it, we observed that there was a considerable spike in memory during the process of importing an image, which led to the crash. This is because app extensions have a memory limit of 120 MB, which can be fairly easily surpassed when loading a high resolution image. So in this blog post, we’ll talk about our findings from investigating our own code and the steps we took the improve the situation.

There have been plenty of new APIs for and improvements to images with the release of iOS 15 at WWDC 2021 — for example, loading images asynchronously using UIImage.prepareForDisplay(), and using AsyncImage in SwiftUI to load images from a remote URL. These APIs are great additions to UIKit. However — spoiler alert! — we won’t be covering them in this post. If you want to read about that take a look at our Loading Images in iOS 15 post. Instead, we’ll be relying on some of the good ol’ existing low-level APIs.

Investigation

The process of importing an image in the UI is carried out by using ImagePickerViewController, which is a custom subclass of UIImagePickerController.

To investigate the memory spike we observed during this import process, we profiled our app on an actual device using the Allocations and Leaks template of the Instruments application.

First, we had to ensure that our speculations were indeed correct and that the spike in memory was during the image importing process. Using Instruments, we were able to determine the point at which the memory spike occurred, confirming what we thought.

• The first thing we did was ensure there was no memory leaking during the import process. The Leaks template came in handy for this. Please note that Leaks may not always capture all the leaks in your app, so it’s important to audit and verify things when using it. Since we couldn’t find any obvious leaks, we dove into our code and observed the API called during the memory spikes.

• We noticed that accessing the image directly in the form of a UIImage instance from the info dictionary in the imagePickerController(_:didFinishPickingMediaWithInfo:) method of the image picker resulted in a memory spike as the entire image was loaded into the memory. This was no surprise, considering UIImage decompresses the image while loading it into memory — an action which also uses a sizable amount of memory.

• Once the selected image was loaded, it was displayed in the UI to allow the user to crop it. However, one thing to keep in mind is that imported images can have different orientations, and they aren’t always oriented up as we expect them to be in the UI.

• So, we redrew the image with an upright orientation using the image orientation data from the image metadata to avoid running into issues where the image wasn’t rendered as expected. However, this didn’t seem entirely efficient, so we started looking into ways to improve.

The Problem

The next step in the import process is cropping. To crop an image, a user performs these steps:

• Sets a crop rect by dragging the crop handles.

• Confirms the cropping area and selects the desired image quality for importing the image from a preset list of fixed sizes.

This is where we noticed our main problem: While cropping the image, we created a completely new image by redrawing only the cropped area to reset the orientation of the image to be up.

The issue with this approach of creating an image was that redrawing into a new bitmap context meant we were recopying the contents of the original image in memory for the newer image instead of reusing the same image, which would have been more efficient. This issue of duplicating image contents also occurred when we created a new image to correct the orientation.

The Fix

The first memory spike when we load an image into memory is something that’s bound to happen at some point because the image needs to be in memory — at least for rendering, if nothing else. However, redrawing the image to fix its orientation before displaying it isn’t necessary. The fix for this issue is simple:

• Don’t redraw the image.

• Ensure that the image orientation is preserved properly when loading the image.

We also decided to add a mechanism to allow loading downscaled images in the imagePickerController(_:didFinishPickingMediaWithInfo:) method. This was done by using the Image I/O API. These APIs provide us with the flexibility of loading an image of a smaller size if required:

// Your image URL.
let imageURL = info[UIImagePickerController.InfoKey.imageURL]!

var cgImage: CGImage
var uiImageOrientation = UIImage.Orientation.up

// Create an image source instance using the image URL.
let imageSource = CGImageSourceCreateWithURL(imageURL as CFURL, nil)!

if shouldLoadDownscaledImage {
    let options = [
        // Specify the maximum size of the downscaled image.
        kCGImageSourceThumbnailMaxPixelSize as String: 1000,
        // Ask to always recreate the image so it matches the above-mentioned max pixel size.
        kCGImageSourceCreateThumbnailFromImageAlways as String: true,
        // Ask the image to be transformed so you don't have to worry about orientation later on.
        kCGImageSourceCreateThumbnailWithTransform as String: true
    ] as [String: Any]

    // Use the below API when you want to load the image in a particular size.
    cgImage = CGImageSourceCreateThumbnailAtIndex(imageSource, 0, options as CFDictionary)!
} else {
    // Create a `CGImage` loading the image at its full resolution.
    cgImage = CGImageSourceCreateImageAtIndex(imageSource, 0, nil)!

    // Extract the image orientation from the image metadata.
    let imageProperties = CGImageSourceCopyPropertiesAtIndex(imageSource, 0, nil) as! [String: Any]
    let extractedOrientation = imageProperties[kCGImagePropertyOrientation as String] as! Int
    let cgImageOrientation = CGImagePropertyOrientation(rawValue: UInt32(extractedOrientation))!
    uiImageOrientation = UIImage.Orientation(cgImageOrientation: cgImageOrientation)
}


let image = UIImage(cgImage: cgImage, scale: 1, orientation: uiImageOrientation)

In the code above, we’re using the Image I/O API to load the image from the given URL. If shouldLoadDownscaledImage is enabled, we restrict the size of the image to a maximum of 1000 × 1000 pixels. This can be incredibly useful when used in memory-constrained environments such as app extensions or App Clips.

Now, coming back to our main problem — we were redrawing the image when cropping it, which led to duplication of the image contents, thereby doubling our memory usage unnecessarily. This is where an easy-to-overlook Core Graphics API came to our rescue: CGImage.cropping(to:) (CGImageCreateWithImageInRect in Objective-C). We created a transform from the crop rect and applied it to the original image — in this way, we didn’t need to allocate any more memory for the orientation-corrected cropped image. This is because the aforementioned API uses the existing image to perform the cropping operation:

let image = imageView.image
let cgImage = image.cgImage!

// Cropping rect in UIKit's coordinate system.
let cropRect = ...
// Since the image can have a non-up orientation, create a transform that can
// convert a UIKit coordinate to that of the oriented image's coordinates.
// Code not included for the purpose of brevity.
// Uses an identity matrix and applies translations or rotations using the given size and
// image orientation so that the drawing starts from the top-left corner.
let rectTransform = transform(for: image.imageOrientation, imageSize: image.size)

// Use our golden API to create the cropped image.
let croppedImage = cgImage.cropping(to: cropRect.applying(rectTransform))

// Create a `UIImage` to display it into another `UIImageView`.
let image = UIImage(cgImage: croppedImage, scale: 1, orientation: image.imageOrientation)

By making the changes shown above, we were able to significantly decrease the memory usage during the image cropping process from around 90 MB to just around 2 MB!

Below is a rough aggregation of the change in the memory we observed. These benchmarks were carried out using an image with a resolution of 3024 × 4032 pixels on an iPhone 11 running iOS 14.

Conclusion

At PSPDFKit, we care deeply about building software that makes optimum use of resources, which is why we return to various parts of the SDK to make improvements. Image loading and resizing can be tricky to get right, and it’s easy to make mistakes that degrade the user experience of our apps. But often, we can optimize our own code by fixing leaks or removing inefficiencies. For most use cases, high-level image APIs should be enough, but sometimes diving deeper and using low-level APIs like Image I/O or Core Graphics has its own advantages, as they can be quite useful and are incredibly powerful when used correctly.