Why the Selection Changes When You Do Syntax Highlighting in a NSTextView and What You Can Do About It

Nov 29th, 2017

On iOS, this does maybe not happen at all, but when you want to write syntax highlighting code for macOS apps, copying together stuff from around the web, you’ll end up with broken application behavior. In short: when you type and the attributes of the line change, the insertion point is moved to the end of the line. That sucks.

TL;DR: Do not perform style changes inside of NSTextStorage.processEditing() or NSTextStorageDelegate methods. Instead, subscribe to NSText.didChangeNotification, or hook into NSTextViewDelegate.textDidChange(_:) (actually declared in its ancestor protocol, NSTextDelegate.textDidChange(_:))

Description of the Problem

Illustrated in steps, it looks like this when you edit a line at the beginning and change the color of the whole line in accordance to the syntax:

A lot of examples on the web are based off of NSTextStorage subclasses. Others use the NSTextStorageDelegate.textStorage(_:willProcessEditing:range:changeInLength:) callback with similar results. To simplify the explanation, let’s assume you subclass NSTextStorage and override processEditing() to apply syntax highlighting as advertised on objc.io:

class CustomTextStorage: NSTextStorage {
    // ... rest of the text storage methods ...

    override func processEditing() {
        processSyntaxHighlighting()

        // Apply highlighting before super so that attribute changes are combined
        super.processEditing()
    }

    func processSyntaxHighlighting() {
        // Parse text based on `self.editedRange` and apply styles, e.g.:
        let paragraphRange = self.string.paragraphRange(for: self.editedRange)
        if paragraphRange.hasPrefix("# ") {
            self.addAttributes([.foregroundColor : NSColor.red],
                range: paragraphRange)
        }
    }
}

This wireframe applies highlighting to any edited line starting with # and then moves the selection to the end of the line. That means the position of the text caret – or “insertion point”, as the text view calls it – is not where the last user change happened anymore.

This is a widespread problem and the question is often asked on the web; people offer different work-arounds and quick fixes but no-one addressed the source of the problem so far, thus resulting in only 90% correct behavior.

Fixing the Editing Process to not Move the Selection (Warning: All Dead End)

To make sure you see which approaches will not be fruitful (and why), let’s go through a couple of work-arounds.

You can skip these approaches if you want to see the non-failing attempt.

Overriding Private NSLayoutManager API

In a talk from 2012, Max Seelemann of The Soulmen (the creators of Ulysses) showed the audience how to react to this odd phenomenon by applying a post-edit fix, overriding private API in NSLayoutManage._fixSelectionAfterChange(inCharacterRange:changeInLength:). (A video recording of the talk is only available in German.) That does indeed help with the insertion point, but if experiment further, you’ll bump into new display glitches and scrolling issues.

Keep in mind that 2012 was before Ulysses 3 shipped, and it’s 5 macOS versions ago. Maybe this was the only way to fix this behavior of the Cocoa text system back then. Compiling against the latest 10.13 SDK and deploying an application for macOS 10.11 and above, I can safely say that modern apps have no need to resort to this.

This points into the right direction, though: the NSLayoutManager is responsible for changing the position of the caret by altering the selection during processEditing().

Not triggering the NSLayoutManager

You can avoid moving the selection if you do not notify the text system of changes to attributes. That means you do not call NSTextStorage.edited(_:range:changeInLength:) with .editedAttributes in your CustomTextStorage.addAttributes(_:range:) implementation:

class CustomTextStorage: NSTextStorage {

    /// Switch used to prevent style changes in `processEditing` from
    /// triggering callbacks.
    fileprivate var isBusyProcessing = false

    override func processEditing() {
        self.isBusyProcessing = true
        defer { self.isBusyProcessing = false }

        processSyntaxHighlighting() // same as above

        super.processEditing()
    }

    /// The real implementation to delegate changes to.
    let content = NSTextStorage()

    override func setAttributes(_ attrs: [NSAttributedStringKey : Any]?, range: NSRange) {

       // When we are processing, `edited` callbacks will
       // result in the caret jumping to the end of the document, so
       // do not send them!
       guard !isBusyProcessing else {
           content.setAttributes(attrs, range: range)
           return
       }

       beginEditing()
       content.setAttributes(attrs, range: range)
       self.edited(.editedAttributes, range: range, changeInLength: 0)
       endEditing()
   }

   // ... rest of NSTextStorage implementation ...
}

This implementation will apply color changes but not move the insertion point.

It’s a 70% winning solution.

The remaining 30% are non-functional block changes. When you change the style of a block of text that exceeds the line of the insertion point (like a block comment, or something else that spans multiple laid-out lines in the text view), only the current line will be updated. The other lines will have new style information, but the layout manager will not schedule actually drawing them – until you select them with the mouse or move the caret inside with the arrow keys, say.

Invalidate the glyphs in NSLayoutManager to support block changes

So multi-line blocks of text do not redraw with the last approach. When I was so close to success a few weeks ago, I struggled to find anything to force-redraw the rest of the text as needed.

I tried two things:

1. Notify NSLayoutManager about the edited attribute range after processEditing() had finished; and
2. Invalidate a laid-out block of text on screen.

Sending the edited message as usual would, among other things I do not know about, call NSLayoutManager.processEditing(for:edited:range:changeInLength:invalidatedRange:). This is where I tried to hook in, first:

    func processEditing() {
        // Obtain the highlighted text ranges for post-processing:
        let rangesNeedingLayoutFixes: [NSRange] =
            processSyntaxHighlighting()

        super.processEditing()

        // Do the post-processing!!11
        for range in rangesNeedingLayoutFixes {
            for manager in self.layoutManagers {
                manager.processEditing(
                    for: self,
                    edited: .editedAttributes,
                    range: range,
                    changeInLength: 0,
                    invalidatedRange: range)
            }
        }
    }

This does indeed show attribute changes on screen, which means the layout manager generates new glyphs for the whole block as expected.

But when you hit spacebar or delete backwards, in lots of circumstances the scroll view jiggles or bounces up and down.

Another thing I applied to some limited success was to mark a larger range as invalid before changing the attributes:

func processEditing() {
    // Call super first this time
    super.processEditing()

    // Determine the to-be-highlighted ranges before
    let invalidatedRange: NSRange = ...
    self.layoutManagers.forEach { (layoutManager) in
        layoutManager.invalidateDisplay(forCharacterRange: invalidatedRange)
    }

    processSyntaxHighlighting()
}

That works way better but crashes in some cases, i.e. when the user scrolls the text view and you replace the contents during the scroll operation. Yeah, what a weird one. Imagine how long it took to figure out the source of seemingly unrelated, occasional crashes …

Interfere with Attribute Fixing

A suggestion I found on StackOverflow suggested to override NSTextStorage.fixAttributes(in:) and perform highlighting there.

override func fixAttributes(in range: NSRange) {
    processSyntaxHighlighting()
    super.fixAttributes(in: range)
}

This works because attribute fixing seems to be part of the call chain before the NSLayoutManager is notified. If you highlight inside processEditing, the attribute fixing will encompass the highlighted range; if you override fixAttributes, though, you get the user edited range and make sure to only pass that on to super.

Attribute fixing is important to remove inconsistencies in attribute assignments. Overriding the method means that the NSTextStorage now does not fix attributes for the whole range you applied highlighting to, possibly resulting in an invalid representation of attributes. That does not sound good; breaking attribute fixing by itself is bad in principle but might still not hurt much in practice. See “Using Text Kit to Draw and Manage Text” from the docs for more on attribute fixing.

Please note I haven’t tested how bad this approach really is in practice. I found this and dismissed it for good reasons, no need to benchmark anything: Since attributes are not fixed properly, the layout manager will not be notified about the real range of invalid glyphs resulting from the highlighting. The layout manager probably redraw the current line only, and thus have the same issues with long blocks of text like the last failing approach.

Update 2022-09-21: Another thing to keep in mind: the default NSTextStorage will use lazy attribute fixing. For syntax highlighting, this might not be what you’re looking for. I rather cache the computes attributes of the highlighting pass in the text storage’s attributes permanently, and leave attribute fixing to do what it’s intended to do, i.e. clean up stuff and unify paragraph styles. See my overview article for attribute fixing for details.

Understanding the Source of the Problem

Highlighting inside processEditing() will move the caret of the text view. Looking at stack traces (“Why and when does the text view scroll after editing? By how much?”) and comparing parameters passed around in private methods of NSLayoutManager and NSTextView, I came up with the following explanation:

NSLayoutManager receives a combined “edited” messages during NSTextStorage.processEditing() that covers all affected ranges.

It goes like this:

• The user types in the text view. Her change affects only the typed-in character range, resulting in an editMask of [.editedCharacters, .editedAttributes] for the limited range of the edit. In the heading example from above, it would be for an affected character range of NSRange(1..<2) for typing the space character after the hash with a changeInLength of 1. This is just the expected notification.
• The syntax highlighting during processEditing() now interferes. It affects a much broader range, for example the whole paragraph, sending .editedAttributes with a range of, say NSRange(0..<17) in the case of the example line.
• The text storage combines alls calls to self.edited(...), forming a union of the ranges and a union of the editedMasks. The NSLayoutManager receives only a single callback, via .processEditing(for:edited:range:changeInLength:invalidatedRange:) for the larger range of NSRange(0..<17) with [.editedCharacters, .editedAttributes]. Again: the syntax attribute changes are not received separately.
• The layout manager triggers a selection change whenever .editedCharacters is received, calling _fixSelectionAfterChange(inCharacterRange:changeInLength:) which Max had introduced above. The edited character range is meant to be much smaller than the changed attribute range, but we got a combined version only, so that’s why _fixSelectionAfterChange receives the larger range.
• It also triggers NSLayoutManager._resizeTextView(for:) with the whole range instead of the original, much more limited user-edited range. That triggers the bouncing scroll behavior. It’s like selecting and then paste-overwriting the whole block every time you hit a key inside.

Following the process description, you have to avoid expanding the range for the .editedCharacters event when you apply syntax highlighting.

Applying a Real Solution

Once you separate the user-originating text change from the syntax highlighting and end up with 2 layout passes, _fixSelectionAfterChange is only called for the user edit (because it contains .editedCharacters), but not for the attribute change during highlighting. Similarly, _resizeTextView will not be triggered because of style changes.

To end up with two separate layout passes, you have to apply syntax highlighting outside processEditing(). This includes NSTextStorageDelegate callbacks.

The first layout pass is finished when you receive a NSText.didChangeNotification. You can subscribe to it only, or use textDidChange(_:) which is part of NSTextDelegate (and its descendant, NSTextViewDelegate).

Heads up: when you apply attribute changes from outside, make sure to wrap all the highlighting changes in a beginEditing()/endEditing() block or the glyph generation will be triggered immediately for every change, wasting performance.

Here is an approach I find suitable. I don’t use NSTextViewDelegate but the notification mechanism instead because then you can make someone else the delegate if you need one at all. Also, this way the SyntaxHighlighter service is self-contained; if it was the delegate, you could unintentionally disable syntax highlighting by changing the text view’s delegate to another object. Which would leave you with a monster delegate that does everything. Nobody likes that, so here you are:

class ViewController: NSViewController {

    var textView: NSTextView!
    var syntaxHighlighter: SyntaxHighlighter!

    override func viewDidLoad() {
        super.viewDidLoad()
        // ...
        self.textView = // ... or use IBOutlet
        self.syntaxHighlighter = SyntaxHighlighter(textView: self.textView)
    }
}

class SyntaxHighlighter {
    let textView: NSTextView

    private var subscription: NSObjectProtocol

    /// - param textView: The text view which should be observed and highlighted.
    /// - param notificationCenter: The notification center to subscribe in.
    ///   A testing seam. Defaults to `NotificationCenter.default`.
    init(textView: NSTextView,
        notificationCenter: NotificationCenter = .default)
    {
        self.textView = textView
        self.subscription = notificationCenter.addObserver(
            forName: NSText.didChangeNotification,
            object: textView,
            queue: nil) { [unowned self] notification in
                self.textViewDidChange(notification)
        }
    }

    func textViewDidChange(_ notification: Notification) {

        // This implicit type expectation is not Good Code.
        // Prefer to use a NSTextView subclass that codifies this
        // dependency (and offers a `processSyntaxHighlighting` method itself).
        guard let customTextStorage = self.textView.textStorage as? CustomTextStorage
            else { return }

        customTextStorage.beginEditing()
        customTextStorage.processSyntaxHighlighting()
        customTextStorage.endEditing()
    }
}

The notification is, as usual, sent synchronously. So your highlighting code will be executed right after the user hits a key (and after the change itself has been processed) without delay in the same run loop iteration. That means you won’t have a visible delay.

You can take chances and write asynchronous highlighting code, but that quickly looks weird. In complex situations, edited characters will now not appear after a delay; they will appear immediately. But the style changes are delayed, which in some ways feels even weirder.

I found it favorable to tolerate delays in synchronous highlighting that is noticeable (probably above the 100ms response threshold) which indicates: something is working hard here. Having immediate textual feedback but delayed style changes felt like the highlighter was too lazy to do its job properly. Especially so when you abort pending highlighting requests when the user types too quickly.

So that’s it, the most comprehensive explanation of the problem of syntax highlighting in NSTextViews affecting the caret location – and the only working solution that’s not just a half-assed work-around that looks good in demos but fails tests for production-level quality.

Feel free to share other half-baked solutions that kind of worked but eventually let you down in the comments and I’ll expand the list!