Extending the Scribble User Experience
Starting with iPadOS 14, Apple introduced a new feature called Scribble for Apple Pencil users. Scribble allows you to directly start writing into text fields or text views using Apple Pencil, and it’ll automagically convert your handwriting into text.
This feature remarkably improves the user experience of entering text with Apple Pencil. Previously, you needed to switch contexts and put down the pencil to enter text using either a software or hardware keyboard, but this is no longer necessary. In fact, you don’t even need to tap on the text field — you can start writing and Scribble will automatically figure out which text field you intend to write into.
Best of all, developers don’t need to write a single line of code to get the benefit of Scribble in their apps — any editable view that implements the UITextInput protocol gets Scribble for free! That means if you use the ubiquitous UITextField or UITextView, you don’t need to do anything special to get your app ready for Scribble.
Apple has also added several APIs to let developers customize aspects of the Scribble experience: UIScribbleInteraction, UIScribbleInteractionDelegate, UIIndirectScribbleInteraction, and UIIndirectScribbleInteractionDelegate. In this blog, we’ll see how to use some of these APIs to extend the Scribble experience and let users write directly on a view that isn’t a text field or a text view.
Toggling Scribble Support
As I mentioned earlier, the best part about Scribble is that it’s enabled by default in iPadOS 14. However, users can disable Scribble directly from the Settings app if they choose.
If Scribble is enabled, developers also have the option to suppress it for specific text inputs if they so choose. By implementing the scribbleInteraction(_:shouldBeginAt:) method of UIScribbleInteractionDelegate and returning false, you can disable scribbling at a specific location:
func scribbleInteraction(_ interaction: UIScribbleInteraction, shouldBeginAt location: CGPoint) -> Bool { if /* User is not allowed to scribble at location */ { // Disable scribbling. return false } // Allow scribbling. return true }
This is useful for special interactions in your app, such as when Apple Pencil is used for drawing on the canvas. When suppressing Scribble like this, Apple suggests providing a UI to the user so that they can toggle between different modes of pencil interaction.
A Use Case for Scribbling Anywhere
In our PDF Viewer app, we allow users to annotate PDF files. You can add a free text annotation, which is a floating text box that allows text entry. Once you’ve opened a file in PDF Viewer, you’d roughly follow these steps to create a new free text annotation:
-
Tap on the annotation toolbar icon.
-
Tap on the free text annotation icon. This will take you into a mode where you can insert a free text annotation on the page by tapping on any empty spot on the page.
-
Tap on an empty spot on the page. A free text annotation will be inserted where you tapped, and the keyboard will pop up.
-
Type the text you want to enter.
If you’re using an Apple Pencil and iPadOS 14, the keyboard won’t show up by default in step 3, but once the free text annotation has been added, you can scribble on it as shown below.
If you want to add more free text annotations, you need to perform steps 2, 3, and 4 again. Here we have an opportunity to streamline the UX a bit, and instead of making the user tap on an empty space, we could allow them to start writing directly!
UIIndirectScribbleInteraction allows us to implement just that. Here’s what Apple has to say about it:
An interaction for using Scribble to enter text by writing on a view that isn’t formally a text input.
In our app, Scribble also works by default in form fields. In this case, the user needs to first tap on a form field and can then start scribbling on it.
UX Expectations
Before we start writing any code, let’s take a minute to write out our expectations for the UX.
-
When in free text annotation editing mode, the user should be able to write anywhere on the PDF page, and a free text annotation should be created at the location where the user entered their text.
-
After a free text annotation is created, we shouldn’t exit the annotation insertion mode. The user should be able to continuously write and add more annotations if they so choose.
-
If there’s already a free text annotation near where the user is writing, we’ll assume they want to append to the existing annotation.
-
When we’re not in free text editing mode, Apple Pencil should work as before and Scribble should not hijack any touches.
With that out of the way, let’s jump to the implementation details!
Supporting Indirect Scribble Interaction
The first thing we need to do is add UIIndirectScribbleInteraction to our container view that shows the PDF page:
let indirectScribbleInteraction = UIIndirectScribbleInteraction(delegate: self) containerView.addInteraction(indirectScribbleInteraction)
We need to conform to UIIndirectScribbleInteractionDelegate and implement its methods. This protocol identifies each interactable element using an ElementIdentifier. We also need to manage these identifiers for our annotations and the container view. Luckily, PSPDFAnnotation already has a name property, which we can use to uniquely identify each annotation. However, for the container view, we’ll create an identifier:
// Used to identify the container view. let containerViewID = UUID()
Finding Elements and Frames
Let’s look at the implementations for all of the delegate methods, starting with two of the methods used to find elements and frames:
This method is called when a user starts to write something in containerView. Here, we need to return the elements that overlap or that are near the rect where the user is trying to write. Returning the containerView’s identifier will allow the user to scribble on the entire page. If we don’t return any element identifiers in the implementation of this method, Scribble won’t start at all since there’s no element to write into. If there are multiple overlapping elements, we’ll put all of them into the array — in this case, the last element in the array will be considered the topmost element.
Here’s how the implementation of this method — along with a couple of helper methods — looks:
/* Variable used to store the last rect that Scribble requested. It's used to check whether or not an element is currently focused. */ var lastRequestedScribbleRect: CGRect! /* Helper function that lets us know if we're in free text annotation insertion mode. We enable indirect scribbling only if we're in that mode. */ var isFreeTextAnnotationToolActive: Bool { let annotationStateManager = controller.annotationStateManager return annotationStateManager.state == Annotation.Tool.freeText && annotationStateManager.variant == nil } func indirectScribbleInteraction(_ interaction: UIInteraction, requestElementsIn rect: CGRect, completion: @escaping ([String]) -> Void) { /* When not in free text annotation insertion mode, we still want to return the free text annotation that's currently the first responder so that normal Scribble interactions work reliably. */ if !isFreeTextAnnotationToolActive { if let activeFreeTextAnnotation = /* Find the text annotation that's currently being edited. */ { completion([activeFreeTextAnnotation.name]) return } /* No annotation was being edited. Scribble should be suppressed. */ completion([]) return } /* We use this rect later (in `isElementFocused`) to check whether or not the element is focused. */ self.lastRequestedScribbleRect = rect var elements = Array<String>() // If a free text annotation overlaps with rect, add its name to the array. let freeTextAnnotations = /* All free text annotations on the current page. */ for freeTextAnnotation in freeTextAnnotations { let boundingBox = /* Free text annotation's bounding box in view coordinates. */ if boundingBox.intersects(rect) { elements.append(freeTextAnnotation.name) } } /* If there are no free text annotations in the array already, we add the annotation container view's identifier so that Scribble can work on the entire page. We also deselect any previously focused annotation views. */ if elements.count == 0 { elements.append(self.containerViewID.uuidString) // Also deselect any selected annotations. } completion(elements) }
The implementation of this method is much simpler. Like the name suggests, all we need to do in this method is return the frame of the element corresponding to elementIdentifier. One thing to keep in mind here is that the frame we return in this method should be in the coordinate system of the indirect Scribble interaction’s view. So in our case, we need to convert the annotation’s frame from PDF coordinates to view coordinates:
func indirectScribbleInteraction(_ interaction: UIInteraction, frameForElement elementIdentifier: String) -> CGRect { if elementIdentifier == self.containerViewID.uuidString { return containerView.frame } else { if let freeTextAnnotation = /* Find the free text annotation corresponding to the given identifier. */ { let rect = /* Free text annotation's bounding box in view coordinates. */ return rect } return CGRect.zero } }
Managing Focus
Now, let’s move on to the implementation of the delegate methods used to manage focus.
This method asks us whether the element referenced by the given identifier is focused or not. We’ll use lastRequestedScribbleRect here to check whether or not the current element overlaps it. If it does, that means the user is writing near the referenced element and we’ll consider it focused. A simpler way to check for focus is to look at the isFirstResponder status of the element’s text field:
func indirectScribbleInteraction(_ interaction: UIInteraction, isElementFocused elementIdentifier: String) -> Bool { // `containerView` is never considered focused. if elementIdentifier != self.containerViewID.uuidString { if let freeTextAnnotation = /* Find the free text annotation corresponding to the given identifier. */ { let boundingBox = /* Free text annotation's bounding box in view coordinates. */ // Consider the element focused if its bounding box intersects with `self.lastRequestedScribbleRect`. if boundingBox.intersects(self.lastRequestedScribbleRect) { return true } } } return false }
The focusElementIfNeeded method is called when an element needs to be focused by Scribble. As a response to this callback, we need to make the UITextInput for the corresponding element a first responder. Here, we have several different cases to consider. When a user writes near an already existing annotation, we’ll get a call to focus the annotation. If the user writes on containerView, we need to make a new annotation and focus its corresponding UITextInput. If this delegate method gets called and we’re not in free text annotation insertion mode, it means the user is editing an already existing annotation and is trying to scribble on it. For that case, we find the first responder and return it. Here’s how the code looks:
func indirectScribbleInteraction(_ interaction: UIInteraction, focusElementIfNeeded elementIdentifier: String, referencePoint focusReferencePoint: CGPoint, completion: @escaping ((UIResponder & UITextInput)?) -> Void) { /* When not in free text insertion mode, if any annotation's text view is currently the first responder, we pass it along to Scribble. */ if !isFreeTextAnnotationToolActive { if let firstResponderTextView = /* Find the first responder text view. */ { completion(firstResponderTextView) return } // No free text annotation was being edited, so we return. completion(nil); return } // Deselect any previously focused annotations. var textView: UITextView! // User scribbled on an empty part on the page. Create a new annotation. if elementIdentifier == self.containerViewID.uuidString { let freeTextAnnotation = /* Create new annotation at `referencePoint`. */ /* Add free text annotation to document. */ /* Start editing text annotation. */ textView = /* Get the free text annotation's text view. */ // User scribbled near an already existing annotation. Select it. } else { let freeTextAnnotation = /* Find free text annotation with the current identifier. */ textView = /* Get the free text annotation's text view. */ } // Make first responder. textView.becomeFirstResponder() completion(textView) }
Here’s the last method related to focusing that we’ll implement. As its name suggests, this method asks us if we want to delay focusing an element. It’s useful for cases where we want to wait for a bit to let the user scribble before we go ahead and insert a new text annotation. The implementation for this method is simple — if the element is containerView, then we delay focus. Otherwise we don’t:
func indirectScribbleInteraction(_ interaction: UIInteraction, shouldDelayFocusForElement elementIdentifier: String) -> Bool { /* When writing on the annotation container view, we want to delay the focus so that the free text annotation gets created after the user finishes writing. */ return elementIdentifier == self.containerViewID.uuidString }
That’s it!
Other Methods
You might have noticed we haven’t implemented all the methods of UIIndirectScribbleInteractionDelegate yet, but we don’t really need to. Here are the other two methods:
They can be used to find out when handwriting begins and ends for any specific element. We don’t have any use case for these two methods, so we won’t be implementing them.
Scribble in Action!
Now that we’ve finished implementing these methods, let’s take a look at the updated UX!
We were able to skip the step where the user needs to tap on the page before they can start annotating! Now they’re able to write on the page directly and create multiple free text annotations. When the user taps on an empty space, we’ll insert a free text annotation there and end the insertion mode, or the user can manually tap on the free text annotation button on the annotation toolbar to get out of the insertion mode.
Notice that scratching a word to delete it also works on the annotations, and so do all the scribble interactions, e.g. drawing a vertical line to split or merge a word, or circling a word to select it. With the updated UX, it’s a much more streamlined process to add and edit free text annotations.
Conclusion
Scribble is an exciting addition to iPadOS, and it has really changed the way users can interact with Apple Pencil. By implementing the new Scribble APIs in the appropriate places, we can make our apps feel more fluid and we can greatly reduce the friction of using Apple Pencil.
This blog post gives an idea of how to implement Scribble APIs for a specific use case. You should also check out Apple’s official example, Customizing Scribble with Interactions, as well as the amazing WWDC session on Scribble. All of these Scribble improvements are available in PSPDFKit for iOS starting with version 10.1. Click here to try out our SDK.
