Today, I propose a microformat for the open web: the /now Page. Let’s call it h-now.

A microformat is a convention to markup your content in a way that helps software to identify content. Like addresses for navigation, contact cards for quick import into contact files via browser extensions, or events for calendar integration.

In a regular website full of custom HTML markup, aggregators can scan for these microformat blocks and extract relevant information. That’s why a vCard parser can find my contact info.

Information on demand is way more useful than information that demands our attention at moments when we’re not interested.
—Seth Godin, Bulletins vs bulletin boards

And a “/now” page is a bulletin board-like page that indicates what someone is up to. It’s a status page. Unlike status feeds on social media, this page does not show incremental updates. It always show the latest status, period. And it doesn’t do push notifications. You have to go there and look. I’ve got one. There’s a whole index of people with /now pages, run by Derek Sivers, who started all of this.

“The /now page movement” started in 2015, so the idea has been around for a while, and some people have adopted this curated status update convention in the meantime. There are free serviced to start your own page without a web server: thenow.page and Our/Nows, for example.

A problem I see is consumption. I sometimes look at what Matt Gemmel is up to, for example. I have to visit each person’s website manually.

As a mad software developist, I’d like to aggregate the information.

• Input: a list of /now page URLs.
• Output: a single page with all the people’s /now pages, concatenated.

Parsing a regular website to not show the navigation, footer, sidebar, and such things isn’t trivial. I want to narrow down to the actual /now page content. Some folks mark up their content in semantic <article> tags and the content extraction could start from there. But others rely on nested <div>s or whatever, and you have a hard time separating the wheat from the chaff.

That’s where microformats come in. You can maintain the existing look and HTML of your website. In there, you just add a couple of extra information to tell a /now page parser what to look for.

My proposal

People with /now pages, brainstorm together and refine this, please!

Here’s my initial attempt.

Revision 1

We already have the h-entry microformat to markup “episodic or datestamped content”. We can think of this microformat as a way to markup blog posts or tweets in a timeline. This is a good fit already!

If RSS feeds aggregate blog posts in a timeline, a /now page is itself a feed of 1 item. So you can think of it as a feed with exactly 1 h-entry.

With a single page, we don’t actually need to wrap this h-entry in a h-feed, but that’d be fine with me. That’d be an add on for /now page consumers and aggregators. It’s not needed to make this all work, but if it’s there, might as well take the author’s name, URL, and photo from h-feed.

Here’s an example of an h-entry from the microformats wiki that works well as a blog post:

<article class="h-entry">
  <h1 class="p-name">Microformats are amazing</h1>
  <p>Published by 
     <a class="p-author h-card" href="http://example.com">W. Developer</a>
     on <time class="dt-published" datetime="2013-06-13 12:00:00">13<sup>th</sup> June 2013</time></p>

  <p class="p-summary">In which I extoll the virtues of using microformats.</p>

  <div class="e-content">
    <p>Blah blah blah</p>
  </div>
</article>

Add h-now to the h-entry and there you go. Note that there are no “officially” mandatory properties, so we can ditch the p-name, for example. An excerpt of Derek Siver’s /now page can look like this:

<article class="h-entry h-now">
  <div class="e-content">
    <h1>What I'm doing now</h1>
    <h2>Home routine</h2>
    <p>I'm in a little 2-bedroom house in Oxford, England, with my son, 
       where we're drawing, reading a lot, playing frisbee, making Lego, 
       walking in the nearby woods, practicing math, and more. All of it offline.
    </p>

    <hr/>
    <p>Updated 
       <time class="dt-published" datetime="2020-05-27">May 27th, 2020</time>,
       from <span class="p-location">Oxford, England</span>.
    </p>
  </div>
</article>

Please note that I had to encompass the whole text content in a <div class="e-content">, because otherwise existing h-entry parsers won’t know where the text is.

Copy and paste the HTML from above into this microformats parser to see that everything important is recognized already.

I personally don’t (yet) care much about metadata. The update date is nice. Location might be interesting to know where your digital nomad friends are at the moment.

The content is what’s actually important. So with minimally invasive changes, this can word.

If we ditch the h-entry heritage, we can ditch the e-content part, get rid of the additional <div>, and just make h-now parsers gulp up whatever is inside a h-now block:

<article class="h-now">
  <h1>What I'm doing now</h1>
  <h2>Home routine</h2>
  <p>I'm in a little 2-bedroom house in Oxford, England, with my son, 
     where we're drawing, reading a lot, playing frisbee, making Lego, 
     walking in the nearby woods, practicing math, and more. All of it offline.
  </p>

  <hr/>
  <p>Updated 
     <time class="dt-published" datetime="2020-05-27">May 27th, 2020</time>,
     from <span class="p-location">Oxford, England</span>.
  </p>
</article>

Simple as pie!

Where to go from here

That’s all there is to it to make it work. That’s my revision 1. Revision 2 and onward could make things more flexible and pretty. I haven’t talked about tagging or whatever. All that is not important; it’s just … nice.

So we don’t need much to make this work at all.

Everybody and their grandchildren can change a Wordpress page template to include the h-now class somewhere around the body text. If you run a static site generator, it’s probably even simpler.

A tool for the job: the NowPageAggregator

All of this wouldn’t be very interesting without a working demonstration. So I whipped up a script that is as dumb as it gets: you pass it a list of /now page URLs to it, and it assembles a list of the content. If there’s no h-now markup, things can look a bit wonky as the script applies simple heuristics to determine a container element for the content that usually excludes navigation and such things.

Get it on GitHub at https://github.com/ChristianTietze/NowPageAggregatorPHP

It’s ugly, but it’s functional to demonstrate my point. You can quickly download and test-drive it on your local machine by using the built-in PHP development server (php -S).

Now do something cool with this, and if you have a /now page, make sure to add the h-now class for future aggregators to have an easier time!

In the Zettelkasten sub-universe I hang out, people begin to bring up more and more public thinking places, personal wikis, shared Zettelkästen, and what people seem to now call digital gardens.

I recall the metaphor of the garden and the stream by Mike Caulfield from 2015, where “stream” is ephemeral stuff, like timelines on Twitter and Facebook or blogs (!), and “garden” is curated, cared-for stuff, like a wiki perhaps, where things grow and stay.

I think the digital gardeners of today are living in a direct succession of this, though I cannot track back any true “beginning” of this movement.

One of the digital gardeners people refer is Joel Hooks, who wrote “🌱 My blog is a digital garden, not a blog”. There, he’s talking about the indie biz coach/consultant Amy Hoy:

[How the Blog Broke the Web] is a direct discussion of this idea of sorting posts by dates and how it effectively ruined the best parts of the internet. We’ve moved away from hand crafted home pages that required us to curate and present our best content in the best light.

The summary of Hoy’s post makes a point similar to Caulfield’s piece, but more pronounced: the wide-spread adoption of the blog format killed gardens. The dichotomy is the same; here, we also have a causality of demise.

I link to these resources because I think they all have something important in common: in order to make a website project useful in the long term, a blog alone won’t cut it. A blog makes atomic pieces of content findable and linkable, but there’s no inherent structure. You can go back and edit old blog posts to cross-reference new things, but I find the process to be awkward.

My own attempts to remedy the problem of important pieces getting lost in the stream of newer posts are very humble: I create overviews outside the blog’s timeline. For example the one about the Cocoa Text System or all the stuff I know about using FastSpring.

I experimented with a wiki in the distant past, but found it wasn’t producing as much value for readers as I had hoped. It’s not enough to just sort and link to all your blog posts. You need additional, curated, long-lived content to make wiki pages work. I shifted my focus to push actual app development of my indie business over expanding my presence as an online writer, and re-integrated the wiki drafts into the rest of this page. (See this one on MVVM.)

The wiki is the ultimate form of a garden. It’s the best fit for a hypertext that is meant to be read as a hypertext. Most blogs consist of isolated items with a high recency, but low (long-term) relevancy that are consumed on their own, like news pieces. That’s why Shawn Blanc advised us all to favor relevancy and reduce clutter in the post I just linked to. That can make a blog archive more useful in the future. But it’s still a blog, where the date and time of publication dominate the structure of things. It’s a flat structure, it’s a timeline, and not a deeply nested web.

So maybe the recent discussion about digital gardens spark your imagination to re-think your own approach to maintaining a website! If I had continued my abysmal first wiki drafts from 2017, things would surely have grown thus far :)

This is an overview of a short series of posts I wrote about TableFlip’s 2020 bug. People were getting a “You don’t have permissions” error, a Sandboxing error, when they worked with .txt files, but not with .md files, even though the app treated them the same.

The issue boiled down to a misconfiguration on my side that went by unnoticed until I enabled Sandboxing. I was wrongly defining document types in the app’s Info.plist, and I was reporting supported file types in the NSDocument subclass in a manner that didn’t work out well.

Here’s the posts from the series:

1. My observations and initial sense-making of The Curious Problem of TableFlip Changing the File Extension brought the problem to my attention that NSDocument wants to change the file extension during the save operation. But only if an external editor had changed the file in the meantime. Was it because the original .txt extension lost to the canonical .md extension that I declared in my app’s document types listing?
2. Related, but not helpful: I discovered how to declare two file types as “related” in the Sandbox, for example a movie file and its subtitles files, or the transition from RTF to RTFD (with images) and back. That didn’t solve the underlying problem, but that’s what you’d be looking for if your intention is to change the file type. (I wanted to keep the file type and prevent automatic type changes.)
3. When I was sorting out effects of the overlapping file types for plain text and Markdown, I found the culprit: the file extension changed because I passed "net.daringfileball.markdown" to the NSDocument.revert method, even though the fileType property was set by the system to "public.plain-text". So that’s where the file type and thus the file extension effectively changed. Allowing "public.plain-text" opened another can of worms, though …
4. … namely that the NSSavePanel‘s default format picker has an empty selection. That’s because the supported formats list does not contain a match for "public.plain-text". I found a hack/quick fix to programmatically select a format in the NSSavePanel’s accessoryView. That’s just a band-aid, but its better than the default behavior.
5. In another bonus episode, I show you how to create your own file format picker as a NSSavePanel.accessoryView from scratch. This didn’t help. It was a quote fun unquote exercise, though, and since there isn’t much documentation for this on the web, I’d rarther share what I found than not telling anybody.
6. Finally, we have a look at a different approach: recognize plain text documents as a separate file type and treat it separately in code everywhere, but hide the format from the user when saving documents.

All previous installments in this series were written some time during the past 6 weeks or so. I couldn’t spend a lot of consecutive time on eitherthe coding problem or the posts, so it became a pain to keep up with all the details and experiments – I hope I have nevertheless told a somewhat intelligible story so far.

But 6 weeks ago, my original post to document what was going on went in a totally different direction, and I didn’t publish it so far. The things that I did publish, I discovered much, much later in the process, and in a different order. It makes a lot more sense to tell the story in the order that I told it, though.

With everything we’ve learned so far, the stuff I struggled with at first make a lot more sense now. Here’s follows a revised description of what I initially tried but ultimately found useless once I figured out the rest.

Recap of the final quick fix

In the last episode, I started with a NSSavePanel accessoryView that didn’t auto-select a file format, and finished with a quick post-fix to make a selection for my particular case.

This is the code, for reference: I hook into the prepareSavePanel callback of NSDocument to select the first item of the file type NSPopUpButton if nothing is selected at all:

asd

extension MyAmazingDocument: NSDocument {
    override func prepareSavePanel(_ savePanel: NSSavePanel) -> Bool {
        guard super.prepareSavePanel(savePanel) else { return false }

        // Adjust the save panel configuration:
        savePanel.isExtensionHidden = false
        savePanel.allowsOtherFileTypes = true

        // Find and fix the format picker:
        if let accessoryView = savePanel.accessoryView,
            let button: NSPopUpButton = findSubview(
                in: accessoryView,
                where: { $0.action == #selector(NSSavePanelFileTypeDelegate.changeSaveType(_:)) }) {

            if button.selectedItem == nil,
                // We can safely assume this is non-nil due to 
                // the `where`-clause above.
                let action = button.action {

                button.selectItem(at: 0)

                // Fake a user interaction here to execute the
                // cascade of NSSavePanel side effects
                DispatchQueue.main.async {
                    NSApp.sendAction(action, to: button.target, from: button)
                }
            }
        }

        return true
    }
}

Note I had to defer triggering the usual side effects of a user-based selection. I could leave these lines out and would not perceive any change in behavior, but since I don’t know what usually happens behind the scenes in an NSDocument, it’d be inconsistent to change the selection without triggering the callbacks, I think.

My original, different approach

The following things I tried way, way before I realized that my calls to revert(toContentsOf:ofType:) were the reason things broke down.

Originally, I came to the conclusion that the plain text format is baked into the system and that macOS refuses to let me register the .txt file extension for my own Document Type. You couldn’t have both, it seemed.

So I didn’t want to fight the framework. I didn’t want to continue tryin to treat essentially 3 document types (plain text, Markdown, and CSV) as 2 (Markdown and CSV). It apparently didn’t work well so far.

Consequently, I added a third document type, “Plain Text”, to cover the public.plain-text UTI, and then handled it separately. In all the places where my NSDocument subclass had to know if it supported writing as CSV or Markdown at all, adding Plain Text as another trigger for the Markdown path wasn’t that much of a problem.

Add the Plain Text UTI document type

First, add another document type to the Info.plist to register the app to recognize the plain text format:

<key>CFBundleDocumentTypes</key>
<array>
    <dict>
        <key>LSItemContentTypes</key>
        <array>
            <string>public.text</string>
            <string>public.plain-text</string>
        </array>
        <key>CFBundleTypeName</key>
        <string>Plain Text</string>
        <key>CFBundleTypeRole</key>
        <string>Editor</string>
        <key>LSHandlerRank</key>
        <string>Alternate</string>
        <key>LSIsAppleDefaultForType</key>
        <false/>
        <key>LSTypeIsPackage</key>
        <integer>0</integer>
        <key>NSDocumentClass</key>
        <string>$(PRODUCT_MODULE_NAME).MyAmazingDocument</string>
    </dict>
    <!-- ... all the other document types here ... -->
</array>

Handle the new document type in code

I then added another file type in code, where all known UTIs and file extensions are defined as decision helpers for the NSDocument.

public enum DocumentType {
    case plaintext // <- new!
    case markdown
    case csv

    public var canonicalTypeName: String {
        switch self {
        case .plaintext: return "public.plain-text"
        case .markdown: return "net.daringfireball.markdown"
        case .csv: return "public.csv"
        }
    }
}

The existing configuration of Markdown documents had to be changed, too, so it wouldn’t overlap with plain text anymore. With my document configuration types, that resulted in code like this:

enum DefaultDocumentTypeConfigurations {
    static let plaintext = DocumentTypeConfiguration(
        documentType: .plaintext,
        fileType: DocumentTypeConfiguration.FileType(
            reading: [
                "public.plain-text",
                "public.text"
            ], writing: [
                "public.plain-text",
                "public.text"
            ]),
        extensions: [
            "txt"
        ])

    static let markdown = DocumentTypeConfiguration(
        documentType: .markdown,
        fileType: DocumentTypeConfiguration.FileType(
            reading: [
                "net.multimarkdown.text",
                "net.daringfireball.markdown"
            ], writing: [
                "net.multimarkdown.text",
                "net.daringfireball.markdown",
                "pro.writer.markdown"
            ]),
        extensions: [
            "md", "mdown", "markdown", "mmd", "text", "txt"
        ])
}

With this code in place, I now could distinguish files depending on their NSDocument.fileType and UTIs. Some resolve as DefaultDocumentTypeConfigurations.plaintext, some as .markdown. In practice, both cases were handled the same, but keeping them apart helped to get rid of the original error where the file type and thus the file extension would change.

It is a clean separation of what the system considers to be separate types.

This originally made me very happy already, because it looks like this is a way to properly model the reality in code. The system thinks that plain text files are special because they have a well-known UTI? Fine, then we’ll treat them in a special way in code, too.

Unintended consequences when adding new document types

As far as I know, there’s no way to tell the NSSavePanel to not include some file formats in its format picker accessoryView when NSDocument.shouldRunSavePanelWithAccessoryView is true (the default). It simply includes all matching document types, and that’s it.

This poses a problem, because with all the changes above users would now be able to select Markdown, CSV, or Plain Text from the file format picker. There’s no such thing as a non-Markdown plain text table file as far as TableFlip is concerned. — I only want to read in .txt files as Markdown documents, remember?

Remove an offending file format from the NSPopUpButton

So I want to support plain text as a document type and use the .txt file extension. But I don’t want the free NSSavePanel.accessoryView to list the document type separately. It should be treated as Markdown. There’s no additional configuration to skip formats in the format picker. What to do?

I could whip up my own accessoryView, but we’re not doing that here. If you want to go that route, you may get the effect of switching file formats for free when you set up your NSPopUpButton‘s target/action mechanism similar to the default accessoryView of NSSavePanel: make it changeSaveType(_:) on your NSDocument and see what happens!

The document type’s name (CFBundleTypeName) is set to “Plain Text” in my Info.plist. When I don’t set it, the popup button will show the UTI, "public.plain-text". So since I set it in the Info.plist, I know with certainty what the popup button’s item title is – it’s not provided by the system. It’s under my control.

Armed with the knowledge of the existing popup button item title and the default target/action setup, it’s simple to find the NSPopUpButton in the accessoryView and remove one of its items:

/// Makes the selector `changeSaveType(_:)` available to Swift.
/// It's used as the popup button's action, but is private API.
@objc protocol NSSavePanelFileTypeDelegate: class {
    func changeSaveType(_ sender: Any?)
}

/// Helper to find a view in a view hierarchy.
fileprivate func findSubview<V: NSView>(
        view: NSView, where predicate: (V) -> Bool) -> V? {
    if let view = view as? V, predicate(view) { return view }
    for subview in view.subviews {
        if let result = findSubview(view: subview, where: predicate) {
            return result
        }
    }
    return nil
}

class TableDocument: NSDocument {
    // ...
    override func prepareSavePanel(_ savePanel: NSSavePanel) -> Bool {
        guard super.prepareSavePanel(savePanel) else { return false }

        savePanel.isExtensionHidden = false
        savePanel.allowsOtherFileTypes = true

        // Find the item's index for plain text in the `accessoryView`.
        guard let accessoryView = savePanel.accessoryView,
            let button: NSPopUpButton = findSubview(
                view: accessoryView,
                where: { $0.action == #selector(NSSavePanelFileTypeDelegate.changeSaveType(_:)) }) 
            let plainTextIndex = button.itemArray
                .firstIndex(where: { $0.title == "Plain Text" })
            else { return true }

        button.removeItem(at: plainTextIndex)
        button.selectItem(at: 0)

        return true
    }
    // ...
}

This is all we need to remove the superfluous NSPopUpButton item.

Enable users to pick file types on their own

You may have noticed that I also customized the save panel a bit more. I changed allowsOtherFileTypes because the .txt extension is a “recognized extension that’s not in the list of allowed types”, as the API docs put it. I apparently cannot bypass this by exporting the Markdown document type with .txt as an allowed extension. The recognition of plain text is just too strong with this computer.

With allowsOtherFileTypes set, you still get a dialog, but at least you get the option to save as .txt, which would otherwise be impossible:

Wrapping up

According to my tests, this hack works fine. You can remove any of the file format items from a save panel without breaking anything.

The worst thing I had feared was that somewhere the index values are used privately, and when you remove an item from the beginning, everything else gets confused. But that apparently is not the case. The private changeSaveType(_:) method must do something else to figure out which file format the user selected – maybe depending on the item’s title. I don’t know.

This leaves us with another hack. You get the file format selector for free, and then you drill it open and change it. I still kinda prefer this over creating your own accessoryView because at least it looks just right.

Is this hack any better than the one I used in the previous episode? There, I modified the default format picker’s initial selection so it wouldn’t show with an empty selection. I’m not sure.

• The empty selection from the other post looks like a bug I introduced that I then steered around; you’re not supposed to not select anything there, so I clearly must’ve configured something wrong. (And yes, I kinda did when I changed the configuration to get to that point!) This feels kind of brittle. But if macOS ever changed to select another item by default whenever the selection would be empty, my auto-selection quick fix wouldn’t kick in.
• Here, I rely on the same mechanism to automatically generate the accessoryView but don’t like what AppKit produces. The result is 100% correct based on the info I pass in. There is no bug, it’s expected behavior. I feel that this is a better foundation than building on a buggy appearance. – My building-upon the foundation means I remove an item from the format picker, though. That’s more invasive than fixing an empty selection.

It’s the plague or cholera. Either have a brittle foundation and use a less invasive fix, or have a solid foundation and outright remove parts of the default UI.

You may choose your own path to hell. I picked mine and went with the selection fix-up.

Maybe one day I’ll revisit this and scratch everything, then recognize Plain Text as a proper document type and implement my own accessoryView to pick file formats, even though I despise how that is supposed to work.

Thanks for tagging along, and godspeed to you and your document-based application development!

NSDocument‘s save panel comes prepared with a file format picker that contains the app’s registered document types. This is a very nice convenience, but when you run into trouble with the default picker, as I did, you wonder if writing your own from scratch would be easier.

It kind of is pretty easy, but tying into the ways NSDocument uses save panels, this is also a bit ugly.

Bring your own accessoryView

You can remove the default accessory view by overriding NSDocument.shouldRunSavePanelWithAccessoryView and return false there. That’s it, no more default accessory view for you.

Your NSDocument subclass is now skipping quite a lot of work in the default implementation of prepareSavePanel(_:). That means you have to do all the heavy lifting in an override-extension yourself.

class TableDocument: NSDocument, NSSavePanelFileTypeDelegate {
    // Tell the system that you bring your own accessory view
    override var shouldRunSavePanelWithAccessoryView: Bool {
        return false
    }

    /// Store a reference of the save panel for later use in `changeSaveType(_:)`.
    /// Can be weak because the callback is supposed to be used while the panel
    /// is visible.
    fileprivate weak var currentSavePanel: NSSavePanel?

    override func prepareSavePanel(_ savePanel: NSSavePanel) -> Bool {
        self.currentSavePanel = savePanel

        guard super.prepareSavePanel(savePanel) else { return false }

        // Add file format selector:
        let popupButton = NSPopUpButton(
            frame: NSRect(x: 0, y: 0, width: 300, height: 22), 
            pullsDown: false)
        popupButton.addItems(withTitles: [
            "Markdown",
            "CSV"
        ])
        popupButton.action = #selector(changeFileFormat(_:))
        popupButton.target = self

        // Set initial selection:
        popupButton.selectItem(at: 0)
        savePanel.allowedFileTypes = self.writableTypes(for: .saveToOperation)

        // (You'll also want to add a label like "File Format:", 
        //  and center everything horizontally.)

        let accessoryView = NSView()
        accessoryView.addSubview(popupButton)
        savePanel.accessoryView = accessoryView

        return true
    }

    @IBAction func changeFileFormat(_ sender: Any?) {
        guard let popupButton = sender as? NSPopUpButton else { return }

        // You need a property to get the `NSSavePanel` instance here;
        // there's no other convenient way, i.e. via the `sender` :(
        guard let currentSavePanel = self.currentSavePanel else { return }

        // (implementation follows right below, please read on)
    }
}

The view layout is pretty simple. I don’t use Auto Layout in this sample code and thus rely on translatedAutoresizingMaskIntoConstraints being true by default so the selector takes up some minimum space in the panel. You can play with this to add a label, position stuff properly, and maybe even transform everything to Auto Layout on your own, I assume.

When you change document types, the save panel will not update the file extension in the file name text field for you though.

To have the file extension change e.g. from .csv to .md when you select another document type, you need to implement this in changeFileType(_:). I observed in the debugger at which point the NSSavePanel.nameFieldStringValue is changed in the default accessoryView callbacks.

— Thanks to Daniel Kennett for pointing out that even though symbolic breakpoints don’t seem to work, maybe because of Sandboxing restrictions where the NSSavePanel runs in another process, KVO on the property still works!

The change to the visible file extension in a save panel’s text field happens when you change NSSavePanel.allowedFileTypes: as a side effect, the panel will update the file extension, unless the user changed it manually. You can read as much in the docs:

If no extension is given by the user, the first item in the allowedFileTypes array will be used as the extension for the save panel. If the user specifies a type not in the array, and allowsOtherFileTypes is YES, they will be presented with another dialog when prompted to save.

Here is an implementation that makes use of this “free” side effect, probably much like the default implementation does:

@IBAction func changeFileType(_ sender: Any?) {
    guard let popupButton = sender as? NSPopUpButton else { return }
    guard let currentSavePanel = self.currentSavePanel else { return }

    switch popupButton.indexOfSelectedItem {
    case 0: // Markdown
        currentSavePanel.allowedFileTypes = ["net.daringfireball.markdown"]
    case 1: // CSV
        currentSavePanel.allowedFileTypes = ["public.csv"] 
    default:
        break
    }
}

This is a very shortened implementation: I leave out how I store the file types in the app. Here, they are just two hard coded cases, tied to the indexes of the popup button’s items. Please apply proper abstractions at home to avoid bugs due to inconsistencies.

And that’s the minimum you need to do to implement your own file format picker for an NSSavePanel.

This wasn’t of any help in my own journey, though: I was struggling with the app changing the file extension from .md to .txt, pulling the rug under its own feet, so to speak, and producing file access errors in its Sandboxed environment. Writing my own accessoryView didn’t help with that.

In my quest to vanquish bugs from TableFlip, we intially observed that file extensions would attempt to change after an external file change. It turnes out that this was my fault because I effectively always passed "net.daringfireball.markdown", even when the document type originally was "public.plain-text". Passing the wrong file type to NSDocument.revert(toContentsOf:ofType:) was causing the issue. Trying to use the optional NSDocument.fileType property first, and then providing a fallback, solved the file extension problem.

It caused another quirk, though. One that I remember from the early days of developing TableFlip: when you would get a “Save As” dialog, the default NSSavePanel.accessoryView displays a file format picker, which I like, but it now doesn’t auto-select the Markdown format there in this case. If the document’s fileType is "public.plain-text", the format picker has an empty selection:

The save operation still works fine. The picker just looks broken. That’s not acceptable, of course.

It appears that this only happens when a .txt file is opened and then “Save As” is used. The auto-selection works perfectly fine when a CSV file is being saved-as, or duplicated, or when a brand new document is saved to disk.

The good news is that the picker shows all the supported document types, nothing more and nothing less. I point this out because back in the day I had a 3rd picker item show with public.plain-text for some configurations.

Sorting all this document saving out is oddly frustrating, even though the documentation appears to be pretty straight-forward. Even after all these years, I still don’t feel confident in my changes right off the bat and have to experiment with tons of things to get a feel for how the components behave. It’s not bad, but it’s weird, and annoying. If you get in a similar situation, know that you’re not alone with this.

We can observe how the save panel is configured in NSDocument.prepareSavePanel:

override func prepareSavePanel(_ savePanel: NSSavePanel) -> Bool {
    guard super.prepareSavePanel(savePanel) else { return false }

    Swift.print(savePanel.allowsOtherFileTypes)
    // => false
    Swift.print(savePanel.allowedFileTypes)
    // => ["net.daringfireball.markdown"]
    Swift.print(self.writableTypes(for: .saveAsOperation))
    // => ["net.multimarkdown.text", "net.daringfireball.markdown", "public.csv"]

    Swift.print(self.fileType)
    // => "public.plain-text"
    Swift.print(self.autosavingFileType)
    // => "public.plain-text"

    return true
}

Why Swift.print? Because just print would call NSDocument.print, invoking the actual document printing dialog.

With these default settings, you cannot pick just any file extension like .fantasy – the app will outright forbid that choice. Here’s the override point to change that.

It’s interesting to see that allowedFileTypes contains just one element, even though writableTypes(for:) returns all possible formats for the current document (CSV is included because the test file only contains 1 table; with 2+ tables, only Markdown would be listed). Also worth noting is that the first element of the document type UTI array is used (the LSItemContentTypes setting), and not the first element in the list of supported file types. So the Info.plist settings clearly win here. When I reorder the elements there, the new first item of the Markdown document type will be picked.

There’s no easy way to access the NSSavePanel‘s format picker. The accessoryView is just any old NSView, and you have to traverse the view hierarchy to find the NSPopUpButton and change its selection.

As a quick fix to put the app into a consistent state, as far as user interactions are concerned, I want to select the “Markdown” document type when nothing is selected and the document itself is in fact a Markdown-compatible document. (Which, incidentally, is always the case. If you have an app with different document type layouts, your mileage may vary.)

Since only .txt files do not auto-select something, I can focus on this one case.

Finding the NSPopUpButton in a NSSavePanel’s accessoryView and selecting an item programmatically

Through debugging, I found out that the NSPopUpButton’s action is changeSaveType(_:). That is private API of NSDocument. This means that Swift cannot find the appropriate symbol when you write #selector(changeSaveType(_:)). Since this action–target-mechanism is from Objective-C-land and uses dynamic dispatch anyway, you can just invent a protocol with a method of a similar signature and use that. It doesn’t matter that the actual implementation doesn’t correspond to the formal protocol definition, because the NSObject’s informal protocol conformance is all what counts – i.e. the fact that -respondsToSelector: returns true when tested for a method with this name.

To find the correct NSPopUpButton, you can filter through all the subviews and select a NSPopUpButton where the action property equals #selector(YourProtocolName.changeSaveType(_:)). That will totally work.

Be aware that this is private API that could have changed throughout the eons. It could change with the next macOS system upgrade for all we know. It’s not exactly Apple’s day-to-day practice to rename methods like this in AppKit, but you should be aware for the potential issues you can run into.

Since this is private API, and since I am certain that the save panel only has one format picker NSPopupButton by default anyway, we can get by without the action == #selector(YourProtocolName.changeSaveType(_:)) predicate.

For completions sake, I’ll include it here, but please exercise caution if you find yourself in a similar situation and consider using this exact code!

// The formalized protocol of the informal private API
@objc protocol NSSavePanelFileTypeDelegate: class {
    func changeSaveType(_ sender: Any?)
}

extension MyAmazingDocument: NSDocument {
    override func prepareSavePanel(_ savePanel: NSSavePanel) -> Bool {
        guard super.prepareSavePanel(savePanel) else { return false }

        // Adjust the save panel configuration:
        savePanel.isExtensionHidden = false
        savePanel.allowsOtherFileTypes = true

        // Find the format picker:
        if let accessoryView = savePanel.accessoryView,
            let button: NSPopUpButton = findSubview(
                in: accessoryView,
                where: { $0.action == #selector(NSSavePanelFileTypeDelegate.changeSaveType(_:)) }) {

            // Fix the empty selection
            if button.selectedItem == nil {
                button.selectItem(withTitle: "Markdown")
            }
        }

        return true
    }
}

/// Find a subview inside `view`'s hierarchy that is of type `V` and satisfies `predicate`.
fileprivate func findSubview<V: NSView>(
        in view: NSView, 
        where predicate: (V) -> Bool) -> V? {
    if let view = view as? V, predicate(view) { return view }
    for subview in view.subviews {
        if let result = findSubview(in: subview, where: predicate) {
            return result
        }
    }
    return nil
}

But beware!

This does change the NSPopUpButton’s selected item, but it does not execute any side effect of a user-originated selection. I display the file extension in this panel. So switching from Markdown to CSV will change the file extension from .md to .csv in the file name text field at the top.

With this kind of programmatic selection change, nothing like that happens. I tried it with the CSV format. The picker selection changes, but the file extension remains at its original state and displays .md. That’s an inconsistency you’d have to look out for.

Apparently, calling NSApp.sendAction(button.action!, to: button.target, from: button) is performing some of the side effects: passing the wrong sender into this call produces a runtime error. So something happens, but not even calling this as another band aid will solve the inconsistency.

I don’t want no more inconsistencies, even if I cannot observe any problems, so here’s a more thorough or “correct” quick fix. It’s still a hack, but it’s a better hack.

extension MyAmazingDocument: NSDocument {
    override func prepareSavePanel(_ savePanel: NSSavePanel) -> Bool {
        guard super.prepareSavePanel(savePanel) else { return false }

        // Adjust the save panel configuration:
        savePanel.isExtensionHidden = false
        savePanel.allowsOtherFileTypes = true

        // Find and fix the format picker:
        if let accessoryView = savePanel.accessoryView,
            let button: NSPopUpButton = findSubview(
                in: accessoryView,
                where: { $0.action == #selector(NSSavePanelFileTypeDelegate.changeSaveType(_:)) }) {

            if button.selectedItem == nil,
                // We can safely assume this is non-nil due to 
                // the `where`-clause above.
                let action = button.action {

                button.selectItem(at: 0)

                // Fake a user interaction here to execute the
                // cascade of NSSavePanel side effects
                DispatchQueue.main.async {
                    NSApp.sendAction(action, to: button.target, from: button)
                }
            }
        }

        return true
    }
}

Yes. DispatchQueue.main.async, aka enqueueing the fake user interaction onto the next run loop iteration (or later), does the trick.

I changed button.selectItem(withTitle: "Markdown") to button.selectItem(at: 0) because with these changes in place, it doesn’t matter if the programmatic selection matches the original suggestion anymore. The save panel will be in a consistent state. Might as well not hard-code the item title in there and just select the first item. (Which, in TableFlip’s case, is the Markdown format. The first element, I’d like to think, is the preferred one anyway.)

This works. Case closed, for now. Up next is a rundown of a different way I tackled this whole problem until I came full-circle and ended up preferring this.

In my initial post about this problem, I talked about the observations and how I began to figure out where the permission problem came from. I turned out to be an attempt at changing the file extension from .txt to .md. When the user opens a .txt file in your app, macOS makes sure you only get access to that exact file path by default. You cannot just write willy-nilly anywhere else without the user’s permission. File extension changes are included in this protection.

So that’s good to know, but why does the app actually try to change file extensions? I guess I botched the document type setup a while back and never noticed until I added Sandboxing way later in the application’s lifetime.

Here’s a rundown of my own setup for failure.

You see, TableFlip opens Markdown files; Markdown files sometimes have the .md or .mmd or .markdown extension or similar, but often come as .txt as well. It is plain text, after all.

Now .txt file are by default reported to be of the Universal Type Identifier (UTI) "public.text" or "public.plain-text". That’s baked into macOS, it seems.

The .md extension and its friends isn’t a known default UTI, so any Markdown editor has to specify the UTI and a list of recognized extensions to help macOS and NSDocument figure things out.

Until today, TableFlip used to have only 2 document types registered: CSV and Markdown. Here’s a tl;dr representation of the stuff you put into the Info.plist in code form:

static let markdown = DocumentTypeConfiguration(
    documentType: .markdown,
    // List of all file type UTIs that are associated with this:
    fileType: DocumentTypeConfiguration.FileType(
        reading: [
            "net.multimarkdown.text",
            "net.daringfireball.markdown",
            "pro.writer.markdown",
            "public.plain-text",
            "public.text"
        ], writing: [
            "net.multimarkdown.text",
            "net.daringfireball.markdown"
        ]),
    // No matter what comes in: what should we treat this as?
    canonicalTypeName: "net.daringfireball.markdown",
    extensions: [
        "md", "mdown", "markdown", "mmd", "text", "txt"
    ])

I use this representation to implement the common NSDocument callbacks to determine writableTypes(for:) and recognize files by file type or extension. You can specify different “reading” and “writing” file types in NSDocument, so I wanted to reflect that here as well.

Way before TableFlip began to write CSV files, it could read CSV files but would only write out Markdown files. Similarly, with the setup above, TableFlip would read plain text files, e.g. .txt, but not write them. When a “read”-operation reported the file type public.plain-text, the app would know that this is supposed to be part of the markdown configuration and store this info locally in the NSDocument object. This DocumentTypeConfiguration instance would then be used when the app is saving changes, e.g. in the save operation callback writableTypes(for:). It would return the array ["net.multimarkdown.text", "net.daringfireball.markdown"] there. Note the absence of public.plain-text.

My interpretation:

1. The app reads a .txt as public.plain-text and sets the file type once; this info stays untouched while you edit the document in TableFlip.
2. But once you perform external changes in e.g. TextEdit, which treats the file as a document as well, and multiple Sandboxes are involved, and access now goes through a NSFileCoordinator behind the curtains, well then apparently TableFlip reassures itself about the writable file types.
3. Notably, the array of writable file types does not include the plain text UTI, so this is treated like a file type change internally, resulting in a file extension change as well.

Where does step (2), the “reassure what file type we’re dealing with”, happen?

In my implementation of the “file changed on disk” callback, NSDocument.presentedItemDidChange, I check if the contents really did change (and ignored events where only metadata would change, as e.g. Dropbox would regularly do). The API docs advise us to do exactly that:

Because this method notifies you of both attribute and content changes, you might want to check the modification date before needlessly rereading the contents of a file. To do that, you must store the date when your object last made changes to the file and compare that date with the item’s current modification date. Use the coordinate(readingItemAt:options:error:byAccessor:) method of a file coordinator to ensure exclusive access to the file when reading the current modification date.

If the user has pending changes, TableFlip asks what to do with them. If the document cache isn’t dirty, though, the new data should be read from disk; when there are no conflicts and the app can update its data to the stuff on disk, I call NSDocument.revert(toContentsOf:ofType:) for this:

guard let fileURL = self.fileURL else { return }
try self.revert(toContentsOf: fileURL, 
                ofType: self.documentType.canonicalTypeName)

Aaaaaaannd that’s where the canonicalTypeName is used. That’s the problem. For Markdown, as we’ve seen above, this produces "net.daringfireball.markdown". And for some reason the default extension (maybe because of precedence order in the file extensions array?), the system changes the file extension when the type name changes from .txt for plain text to .md for Markdown.

So that’s the source of the problem: I was getting a public.plain-text file for reading, stored it internally of type “Markdown”, then upon external (!) changes, reverted the contents and used the cached document type that didn’t know what the original type name really was, thus changing it to net.daringfireball.markdown.

The simplest fix to approach this is rely on NSDocument.fileType wherever possible, which reflects the UTI, and use the canonical type name as a fallback only:

guard let fileURL = self.fileURL else { return }
try self.revert(toContentsOf: fileURL, 
                ofType: self.fileType ?? self.documentType.canonicalTypeName)

With that quick fix to the file extension change problem, I’m left dealing with another issue: when the document type is public.plain-text, the NSSavePanel‘s accessoryView file format picker doesn’t auto-select anything.

I’m telling this story not in a chronological order. I arrived at this point waaaaay later, after I tried many different things, including adjustments to the recognizes document types and customizations to the save panel. So in the next installment, we’ll have a look at that.

When I researched what caused TableFlip to lose permissions to write out files, I learned about “related items” in the Sandbox.

A good example from the docs is to open a movie file plus its subtitle captions together. The user would select the movie in an NSOpenPanel. This usually grants access to the movie file directly per Sandboxing restrictions. You can declare the subtitles file to be a related item so that the Sandbox allows you to read both. That’s clever.

The implementation of item-relatedness is to add NSIsRelatedItemType with the value YES to the related file (e.g. the subtitles) movie document type and caption document type in your app’s Info.plist.

• You can add it in the app target’s “Info” tab under the document type’s “Additional document type properties” list, or manually.
• To add it manually, you have to go to the app’s Info.plist, expand the CFBundleDocumentTypes (“Document Types”) array, expand your items (each is a dictionary), and add the key right there. As of Xcode 11.3, I had to close and re-open the project to have the document type changes be reflected in the app target’s “Info” tab.

This didn’t help in my situation with TableFlip, though, where plain text (.txt) and Markdown (.md) were confused. Marking them as “related” is not a proper way to address this problem, because one does not supplement the other. Of course I tried, and the underlying error, “could not get a sandbox extension”, went away once I added NSIsRelatedItemType to the “Markdown” document type and left the plain text one as-is. That doesn’t make much sense in terms of related-ness, though, and the result were appropriately confusing: while the error went away, the app would now have permission to change the file extension from .txt to .md. But I don’t want the extension to change, so this didn’t address the real problem.

The App Sandbox Design Guide documentation talks about two scenarios where related items make sense. One is the movie + subtitles scenario I already mentioned. The other is when the app needs to “save a file with a different extension than that of the original file.” Their example is when you start with an RTF, insert an image, and now need an RTFD. To handle this case, you both need to specify the relation so you have write access to .rtfd as well, and rename the file properly. This is similar to the result in TableFlip after I added the NSIsRelatedItemType flag, and the rename action is addresses in a proper way, but remember I don’t want the rename to happen in the first place.

I didn’t find a lot of info on the web about this, so there you go, I documented the process for you.

Users have reported problems with TableFlip saving their files recently. One wrote about it in the Zettelkasten forums, if you want to see the problem in context.

To reproduce the problem: when you open foo.txt in TableFlip and a text editor, then change the file in the editor rapidly, TableFlip would show a “You don’t have permissions” error once you tried to save changes from TableFlip later.

That’s a sandboxing error. Since the user opened the file in TableFlip, and since TableFlip is a NSDocument-based app with proper URL bookmark handling, this didn’t make sense at first: where did the permission go? Why do I lose permissions at all? Also, TableFlip did not show this error if you only open the file in TableFlip and save changes there, so the file itself was okay up until the external editor changes something.

So it’s related to external changes. In debug builds, I was able to trigger the “reload on external changes” code path without needing another app open, and it produced the problem even quicker. What did the external changes do that makes the persmissions go bonkers?

TableFlip Changes the File Extension During the write Operation

An error I saw in the Xcode console right before the error was presented:

NSFileSandboxingRequestRelatedItemExtension: an error was received from 
  pboxd instead of a token. Domain: NSPOSIXErrorDomain, code: 1
2020-03-26 13:04:13.691060+0100 TableFlip[19188:445816] 
  -[NSFileCoordinator itemAtURL:willMoveToURL:] could not get a sandbox 
  extension. oldURL: file:///.../Documents/foo.txt, newURL: file:///.../Documents/foo.md

• I never called NSFileCoordinator.itemAtURL:willMoveToURL:, this is part of whatever is build into NSDocument.
• I never instructed the app to change the file extension from .txt to .md. But the move command boils down to a file extension change. That’s interesting.
• Symbolic breakpoints on NSFileCoordinator.itemAtURL:willMoveToURL: are not triggered, so there was no way to see when this happens in the code.

In my NSDocument subclass, I override the write variant NSDocument.write(to:ofType:for:originalContentsURL:) and don’t move anything there; something must be handled wrongly elsewhere. There, it turns out that the incoming url and absoluteOriginalContentsURL parameters are not what I expected, though:

• url is file:///var/folders/62/8k21681d08z9lhq8h433z3rh0000gp/T/de.christiantietze.TableFlip/TemporaryItems/(A%20Document%20Being%20Saved%20By%20TableFlip%202)/foo.md
• absoluteOriginalContentsURL is file:///.../Documents/foo.md

Remember that the file, when I opened it, was file:///.../Documents/foo.txt, not .md. This tells me:

1. TableFlip reads a foo.txt from disk,
2. saves changes atomically using a temporary directory in the sandbox with the .md extension,
3. apparently wants to move the result to the same directory as the original file;
4. since the sandbox granted access to the .txt only, not the parent directory, moving a new file into that directory is not permitted.

That’s apparently what “could not get a sandbox extension” means.

Now with the weird error messages out of the way and their corresponding dialogs popping in my face – why is the file extension changing in the first place?

Is the file type reported wrong? NSDocument.writableTypes(for:) is not called when the extension changes. It’s not called a whole lot when opening an existing file at all. That’s useful to know, since it rules out a source of problems in code already.

There still was one hideous source of error left that I didn’t pay close attention to up to this point: myself.

Find out what Christian discovered when he looked into the Mirror of Truth in the next episode!

Back in January when I drafted this post, I had just discovered Apple’s NaturalLanguage.framework. I still don’t know how powerful it really is, but it’s useful for a very simple task already:

Counting words.

In English and German, I can get pretty accurate results with a String.split at punctuation marks and whitespace. In French, you will get skewed results because these nices folks decided to put whitespace between quotation marks and quoted text.

“Quote in English”
« Citation en français »

Splitting the string at whitespaces will produce 3 for the English line and 5 for the French.

That’s where the linguistic tagging of the Natural Language toolkit comes into play!

It also works for a lot of other languages, even Chinese. (Guess what I’ll change the international algorithm for the WordCounter to.)

To get it to work, create a NSLinguisticTagger with the scheme set to .tokenType: that’s for detecting words and punctuation instead of, say, sentences.

let wordCountTagger = NSLinguisticTagger(tagSchemes: [.tokenType], options: 0)

The actual API to enumerate over all words is more useful for this purpose since macOS 10.13 since you can specify that you’re interested in NSLinguisticTaggerUnit.word directly. With older versions of the OS, you have to tweak the counting a bit and filter matched tags for NSLinguisticTag.word.

Don’t ask me what the purpose of NSLinguisticTag is over NSLinguisticTaggerUnit, and how you can combine these with great success. That still beats me.

func wordCount(tagger: NSLinguisticTagger, text: String) -> Int {
    let range = NSRange(location: 0, length: text.utf16.count)
    let options: NSLinguisticTagger.Options = [.omitPunctuation, .omitWhitespace]
    tagger.string = text

    var count = 0

    // Since macOS 10.13, you can limit the enumeration to words directly:
    if #available(OSX 10.13, *) {
        tagger.enumerateTags(in: range, unit: .word, scheme: .tokenType, options: options) { _, _, _ in
            count += 1
        }
    } else {
        tagger.enumerateTags(in: range, scheme: .tokenType, options: options) { tag, _, _, _ in
            guard tag == NSLinguisticTag.word else { return }
            count += 1
        }
    }

    return count
}

Keep the counter off your main queue

I don’t know how hard linguistic tagging hits your performance. In The Archive, the word counting stats aren’t mission critical; a delay, aka. “eventual consistency”, is totally acceptable. Asynchronous processing keeps the main queue free for important work.

Without RxSwift or other reactive setups, you can start by calling the wordCount(text:) function above from a background queue. It’s generally advised not to use NSLinguisticTagger from multiple threads – but it should be fine to use it from a background queue exclusively. Do exercise caution with all this.

Heads up: You will need a serial dispatch queue. Without a serial queue, results from callbacks may appear out-of-order. For example, if you dispatch the counting on a concurrent background queue with a string that is 10 MiB large, and then immediately dispatch another piece of work with just a couple of characters, the callback for the shorter string will likely finish first. If you display counts for multiple strings in a table, the order doesn’t matter. If you want to abort the previous request and only get back the results for the shorter string, you need a different approach.

With serial dispatch queues, results do come back in order, and a longer running but outdated request blocks execution of later requests. If you always want to display the current word count for a document, for example, cancelling previous requests is the sensible option. For this, you will have to use DispatchWorkItem or come up with your own concoction of a cancellable dispatched block.

Here’s what a work item approach may look like:

let wordCountQueue = DispatchQueue(label: "wordCounting", qos: .background)

/// Keep track of the previous work so you can cancel it.
private _previousWorkItem: DispatchWorkItem?

func wordCount(text: String) { /* Expensive calculation here */ }
func asyncWordCount(text: String, 
                    completion displayCount: @escaping (Int) -> Void) {
    _previousWorkItem?.cancel()
    let workItem = DispatchWorkItem {
        let count = wordCount(text: text)
        DispatchQueue.main.async {
            displayCount(count)
        }
    }
    _previousWorkItem = workItem
    wordCountQueue.async(execute: workItem)
}

In The Archive, a lot of UI code is written based on RxSwift, so I had Rx event streams at hand anyway. I wrapped the counting in RxSwift Observables, quite similar to the following:

struct WordCountViewModel {
    let wordCountTagger: NSLinguisticTagger

    // Like the function from above
    private func wordCount(text: String) { /* ... */ }

    // Input port
    let textChanges: Observable<String>

    // Output port
    var counts: Observable<Int> {
        return textChanges.flatMapLatest { text in
            Single<Int>
                .create { single -> Disposable in
                    // jiggleThread(minimumDelay: 0.5, maximumDelay: 2)
                    single(.success(self.wordCount(text: text)))
                    return Disposables.create()
            }
            // Perform request on other queue
            .subscribeOn(ConcurrentDispatchQueueScheduler(qos: .userInitiated))
        }
    }
}