Decorate NSGlyphStorage to Replace Glyphs On-the-Fly

There are many ways to affect the glyphs that are used to show text on screen. Since macOS 10.11 El Capitan (released in 2015), possibly the simplest override points are in NSLayoutManagerDelegate. Another, arguably more ancient way to replace glyphs on-the-fly is to override and customize NSGlyphGenerator. This predates the existence of NSLayoutManagerDelegate, and it’s not even available on iOS, so that’s how arcane the approach outlines here will be.

I was revisiting WWDC 2010’s session 114, Advanced Cocoa Text Tips and Tricks by Aki Inoue and Dan Schimpf, because I had notes on that session from 2017 in my Zettelkasten when I learned the basics of customizing a text editor. Schimpf/Inoue show how you can implement folding paragraphs in their talk, and they use NSGlyphGenerator to effectively hide all the folded-away glyphs on screen.

In general, NSGlyphGenerator has only one method: generateGlyphs(for:desiredNumberOfCharacters:glyphIndex:characterIndex:). In our subclasses of NSGlyphGenerator, we are supposed to call NSGlyphGenerator.shared which actually provides a useful implementation of mapping characters to glyphs.

In case you’re confused by the terminology: think of characters are the string representation without any info about the looks, and glyphs as the digital, movable type from a font family. Or, for programmers: so actually draw a character on screen you take the character code, map it with a font, then get a glyph that you can draw. More or less.

To actually customize what happens in NSGlyphGenerator, you can decorate the NSGlyphStorage target that is used in the process. This is usually the NSLayoutManager, but the rest of the layout manager is not important for this job. By “decorate” I mean in the “Decorator” or “Wrapper” Design Pattern sense: you implement all required methods, forward all of them to a base implementation, and adjust the parameters you forward and/or the results produced by the base implementation.

In the WWDC session from 2010, Schimpf/Inoue subclassed NSGlyphGenerator and made the subclass itself conform to NSGlyphStorage, so their subclass was actually the decorator. Their delegation to NSGlyphGenerator.shared then passed self – I don’t like that mixing of concepts for didactical reasons.

So even though a bare NSGlyphGenerator subclass is virtually useless, here’s an extracted decorator:

class GlyphGenerator: NSGlyphGenerator {
    private let glyphStorageDecorator = GlyphStorageDecorator()

    override func generateGlyphs(for glyphStorage: NSGlyphStorage,
            desiredNumberOfCharacters nChars: Int,
            glyphIndex: UnsafeMutablePointer<Int>?,
            characterIndex charIndex: UnsafeMutablePointer<Int>?) {
        // Update decorator; the actual NSLayoutManager will probably stay
        // the same and this is a no-op.
        glyphStorageDecorator.base = glyphStorage

        // Call default class-cluster implementation using the decorator
        NSGlyphGenerator.shared.generateGlyphs(
            for: glyphStorageDecorator,
            desiredNumberOfCharacters: nChars,
            glyphIndex: glyphIndex,
            characterIndex: charIndex)
    }
}

final class GlyphStorageDecorator: NSGlyphStorage {
    var base: NSGlyphStorage?

    init(base: NSGlyphStorage? = nil) {
        self.base = base
    }

    func insertGlyphs(_ glyphs: UnsafePointer<NSGlyph>, length: Int,
            forStartingGlyphAt glyphIndex: Int, characterIndex charIndex: Int) {
        guard let base = base else { assertionFailure(); return }
        // TODO: Hide glyphs here :)
        base.insertGlyphs(pointer, length: len,
            forStartingGlyphAt: glyphIndex, characterIndex: charIndex)
    }

    func setIntAttribute(_ attributeTag: Int, value val: Int,
            forGlyphAt glyphIndex: Int) {
        guard let base = base else { assertionFailure(); return }
        base.setIntAttribute(attributeTag, value: val, forGlyphAt: glyphIndex)
    }

    func attributedString() -> NSAttributedString {
        guard let base = base
        else { assertionFailure(); return NSAttributedString() }
        return base.attributedString()
    }

    func layoutOptions() -> Int {
        guard let base = base else { assertionFailure(); return 0 }
        return base.layoutOptions()
    }
}

This is just a very basic scaffolding. The actual glyph replacements have to be done in insertGlyphs.

To hide every 2nd glyph, this would do the trick:

func insertGlyphs(_ glyphs: UnsafePointer<NSGlyph>, length: Int,
        forStartingGlyphAt glyphIndex: Int, characterIndex charIndex: Int) {
    guard let base = base else { assertionFailure(); return }

    var mutatedGlyphs: [NSGlyph] = []
    // Extra fancy: avoid reallocation by preparing the array size
    mutatedGlyphs.reserveCapacity(length)
    for i in 0...length {
        let glyph = (i % 2 == 0)
            ? NSGlyph(NSNullGlyph)
            : glyphs[i]
        mutatedGlyphs.append(glyph)
    }

    mutatedGlyphs.withUnsafeBufferPointer { bufferPointer in
        guard let pointer = bufferPointer.baseAddress else { fatalError() }
        base.insertGlyphs(pointer, length: length,
            forStartingGlyphAt: glyphIndex, characterIndex: charIndex)
    }
}

The glyphs: UnsafePointer<NSGlyph> parameter actually denotes a C-style array using a pointer. We are guaranteed that it has length elements, so we can use Swift’s pointer subscript to get an element from the array at any index using glyphs[i].

You can probably (!) get by when you make use of this pointer to replace elements at particular indexes, but at this point, we don’t own the glyphs array. The calling context does. And who knows what happens when you mess around with the array. That’s why we operate on a copy.

As with many things that predate Swift for 10+ years, the bridging between NSNullGlyph (type Int) and NSGlyph (typealias for UInt32) is a bit awkward. (Why didn’t they declare let NSNullGlyph: NSGlyph = 0?)

Please also note that in the decades since introducing NSNullGlyph, according to the header files NSNullGlyph is soft-deprecated (aka there’s no compiler warning) and we’re supposed to use NSLayoutManager.GlyphProperty.null instead. That is wrapped in a Swift type, and that’s also a strong indicator that NSLayoutManager is the place to go to override glyphs on-the-fly.

All in all, take this piece as a Swift recreation of a historic document that taught Mac programmers over 10 years ago how to affect the stuff you see on screen. And then brace yourselves for what came afterwards. I’ll keep you posted.