The Limits of swift-markdown Customization
So I was happy spending a couple of days working on a Markdown pipeline. One that would allow me to process Markdown files, perform automated tasks, then produce 2 results:
- Generate external files (graph rendiitons in this case, actually), and
- Insert an image reference into the original Markdown file.
The swift-markdown package wraps the cmark CommonMark parser and offers a really nice to use API. Also Marin told me on every other occasion that I should try it, it’s good :)
The parsed Document is itself a top-level markup node (the root of the tree) and you start walking the whole document with a MarkupVisitor that’s either a MarkupWalker (just looking, not returning anything per visit) or a MarkupRewriter (looking and potentially modifying/replacing each visited element).
Now the API works by implementing multiple dispatch:
- You start visiting the (sub-)tree of nodes with
MarkupVisitor.visit(_:), which takesany Markup; so you can start the process with an existential and don’t need to know which type of node is actually visited. - The default implementation of
MarkupVisitor.visit(_:)(in a protocol extension) dispatches back to the visited node viaMarkup.accept(_:)that takes a mutable visitor reference. This is left for each concrete type to implement. - Your
Documentwill receiveaccept(_:), and by this point you know the concrete type. Every element dispatches back to the visitor, but now calling a concrete visitor function that carries strongly typed information – likevisitDocument(_:)in this case. - The concrete
MarkupVisitor.visitDocument(_:)(or any other concrete method with type information relating to your node) does its job. The default implementation from theMarkupVisitor’s protocol extension callsdefaultVisit(_:), which receives an existentialany Markupagain, and in both the walker and rewriter refinement of the protocol apply themselves to the subtree, starting the whole process again for each child node (with the rewriter collecting changed child nodes, while the walker will merely descendInto(_:) and visit elements).
That means you have a 1:1 relation of node types and visitor functions. This is classic double dispatch to achieve runtime polymorphism: You get from a function call on the existential any Markup, via the affected concrete value that knows its type, back to a concrete function that accepts its specific Markup-conforming type.
Sadly that also means you cannot create new node types for your app even if it’s just for producing Markdown string representations with a formatter (so you wouldn’t need to tweak the parser).
Limits of the Package: You Can Only Use What You Get
To introduce new types of markup in the package, and a new visit___ function with it, you need to maintain a swift-markdown fork.
As a user of the package, you do have access to the public protocols to build your own, so it all looks rather promising on a first glance, until you try.
Here’s a concrete example.
In my processing pipeline, I want to insert an image literal  into the file to show the image that was generated from e.g. a code block (imagine a graphviz diagram’s code for example).
The Image node is semantically embedded in its own Paragraph, and it should go after a BlockCode block element. Creating these nodes is straightforward:
Paragraph(
Image(source: filePath, title: nil)
)
With all the visiting and especially rewriting, inserting this into the document tree sounded like a manageable task:
The second refinement, MarkupRewriter, has an associated Result type of an optional Markup element, so it’s meant to change or even remove elements from a markup tree. You can return nil to delete an element, or return another element to substitute in its place. (From the v0.6.0 Docs)
But you can only change one element to another (1:1), or remove it by returning nil (1:0). That’s really it; you cannot prune and graft, splice, inject or whateverify the document tree in any other way (1:N where N>1).
Trials and Dead Ends
To entertain you, dear reader, with my failed attempts, and illustrate the limitaitons of the package for future generations, here’s what you might come up with and try yourself:
Creating Your Own Markup Elements (Dead End)
Now if you can only return 1 element from a rewriter, you could surely replace the affected BlockCode with an element that wraps the BlockCode and the Paragraph with Image? If there’s no “insert element”, only “replace”, then replace it with more!
This is how far you could get:
struct ExtendedElement<Original, Other>: Markdown.BlockContainer
where Original: BlockContainer, Other: BlockContainer {
// 🛑 Type ExtendedElement<Original, Other>' does not conform to protocol 'Markup'
let original: Original
let other: Other
init(original: Original, other: Other) {
self.original = original
self.other = other
}
func accept<V>(_ visitor: inout V) -> V.Result where V: MarkupVisitor {
return visitor.visit(self)
}
}
To conform to Markup, you also need to provide _data in your implementation. But you can’t: while you can “see” _data of type _MarkupData, it’s an opaque value type with initializer and properties only visible to the package itself.
This is purely meant to be parsed, not created on the fly.
In theory, you could create a structure of similar memory layout in your code, then unsafely cast to _MarkupData. In practice, I wouldn’t want to maintain that any more than I would want to maintain a swift-markdown fork. Presented with both choices, I’d prefer the fork.
Using CustomBlock (Limitation)
There’s a block-level element that does nothing, but contais child nodes. It’s called CustomBlock and sounds perfect:
This element does not yet allow for custom information to be appended and is included for backward compatibility with CommonMark. It wraps any block element.
I don’t need custom information (metadata?), I only want a structural replacement. So given the original code block and my to-be-inserted paragraph, I can create a custom block element like so:
CustomBlock(
existingCodeBlock,
Paragraph(
Image(source: filePath, title: nil)
)
)
That’s a 1:1 replacement and I can create a MarkupRewriter-conforming type that produces these when encountering fitting code blocks; simplified:
struct ImageInsertingCodeRewriter: MarkdownRewriter {
let graphRenderer: Renderer
func visitCodeBlock(_ codeBlock: CodeBlock) -> any Markup? {
guard codeBlockCanBeRendered(codeBlock) else { return codeBlock }
let filePath: String = ...
graphRenderer.render(codeBlock, to: filePath)
return CustomBlock(
codeBlock,
Paragraph(
Image(source: filePath, title: nil)
)
)
}
}
And indeed it works, the document is changed from one containing a code block into one containing a custom block with the code block inside! Pruning and grafting the tree works.
But you can’t render these CustomBlock elements to Markdown or HTML; the default formatters will fatalError-out:
Fatal error: Formatter not implemented for CustomBlock
The reason for this is that every concrete visit___ function calls defaultVisit(_:) from the protocol extension of MarkupVisitor. This also applies to formatters, unless they implement their own visit___ functions. MarkupFormatter is used to produce Markdown, and its defaultVisit(_:) produces this error to catch unhandled elements. There’s no visitCustomBlock(_:) for the MarkupFormatter, so it’s an unhandled element and errors-out.
The HTMLFormatter does nothing like that, by the way, and will call the default implementation from MarkupWalker that merely descends into child nodes.
So you need your own formatter.
Luckily, all you need to do is ensure that you walk the whole tree and delegate to MarkdownFormatter except for visitCustomBlock(_:). Here’s a simple version to get started:
struct CustomFormatter: MarkupWalker {
private var baseFormatter: MarkupFormatter
var result: String { baseFormatter.result }
init(baseFormatter: MarkupFormatter) {
self.baseFormatter = baseFormatter
}
mutating func visitDocument(_ document: Markdown.Document) {
// Don't dispatch to baseFormatter here, or it'll walk the whole tree
// and you get the CustomBlock formatter error.
descendInto(document)
}
mutating func visitCustomBlock(_ customBlock: CustomBlock) {
// Transparently skip CustomBlock itself, descending into its
// child nodes directly.
descendInto(customBlock)
}
mutating func defaultVisit(_ markup: any Markup) {
// Forward any other markup to the base formatter.
// (Only works if CustomBlock nodes are limited to be 1st level child
// elements of the document, never nested in e.g. blockquotes, or the
// formatter's traversal kicks in.)
baseFormatter.visit(markup)
}
}
This kind of works, with 2 caveats:
Markuptypes know about their placement in the document.MarkdownFormatterusesMarkup.indexInParentto determine whether a block should ensure 2+ newlines being printed. Even though theCodeBlockisn’t created from scratch (so it has anindexInParent > 0), and the encompassingCustomBlockis transparently skipped, something doesn’t work here and the code block is not separated from previous blocks with any newline. And since we can’t callensurePrecedingNewlineCount(atLeast:)ourselves, I’m stuck here and it boggles my mind. (There’s probably an easy fix I’m overlooking but read on to see why this all doesn’t matter in the end.)- There’s that huge limitations I commented on in the code snippet: that you cannot matching
CodeBlocks in other block-level elements like blockquotes or lists, because once you enter theMarkdownFormatter, its own traversal of sub-trees kicks in and will call its ownvisitCustomBlock(_:), not yours.
So in the long run, you’ll want to write your own formatter to support working with CustomBlocks anywhere. I’ve checked: you can copy and paste the MarkdownFormatter implementation into your code base and it’ll eventually compile fine. There’s no private API involved. That’s a relief: so this would actually be possible.
But if you only want that 1 tweak in your life, you might as well fork swift-markdown and introduce a custom type instead, because maintaining your own MarkdownFormatter implementation is certainly more code than making sure 1 new type and 1 new visit___ function is consistenly used everywhere.
If MarkdownFormatter was designed for customizability, it would’ve been a non-final class. One that you could, you know, subclass and conditionally override methods of.
“But we should prefer composition over inheritance anyway, Christian!” – Even if you don’t abhor the thought of providing 36 visit___ functions that may or may not delegate to MarkdownFormatter, even if you were perfectly willing to pay this price for the sake of composition over inheritance purity, it wouldn’t actually help: As mentioned above, once you delegate to MarkdownFormatter to let it handle a block-level node, it won’t stop processing that part of the document. It’ll descend down the node’s sub-tree itself. It’s a dumb automaton like that. There’s no additional callbacks, no stop signals.
So you actually need your own formatter proper, implementing the serialization yourself.
Then again you’ll want your own formatter anyway, because …
Idempotency Issues
… the default Markdown formatter doesn’t preserve anything from the original input. It completely, well, formats the AST as a new document.
let string =
"""
# Hello
## World
- mixing
+ list
* items
"""
let document = Document(parsing: string)
print(document.format()) /* =>
# Hello
## World
- mixing
- list
- items
*/
If you just re-format an existing document without performing any AST modifications, you’ll change the user’s data. This could be welcome (like an automatic linter), but it could also mess with their personal formatting conventions.
To be fair, the MarkdownFormatter is not designed to be a tool for surgical changes to document files (or in-memory strings). It’s there to produce a canonical, and also somewhat configurable, Markdown document as its output. The formatter type is itself basically a complicated way to write a single function called format with the signature (String) -> String. The fact that it’s a struct, or any type actually, is just an implementation detail that fits Swift the programming language.
With the CustomFormatter I presented above, I shared a working solution to decorate root-level code blocks with an image literal. Even though it’s limited, it does work and demonstrates the idea nicely.
But I wouldn’t want to use a tool like that at this stage of development on my own Markdown documents, let alone have users of my app use it!, because it reformats the whole text as an unintended side-effect.
Writing Custom Delta Renderers
What I do want is a rather simple string insertion: the intended effect is super simple, I want to literally insert something like this after a matching code block:
One like of text with the image literal, and sufficient newlines above and below, that’s it.
I’m fine with replacing, instead of inserting, affected elements (but not the rest of the document).
Ultimately, the Markdown pipeline I have in mind will create an image and inserts the markup on the first run, but on subsequent runs will update existing image references, not insert more and more paragraphs with image literals. To do this, the matcher will need to match more than an isolated CodeBlock: it needs to match the code block and the following paragraph (if any). Then render only this combination and replace the relevant part of the Markdown string.
The part about applying a replacement should be simple, because every Markup element knows its SourceLocation in terms of line number and UTF-8 column offsets, matching cmarks parser output. The MarkdownFormatter ignores these details and just produces a new document. But in theory, you could track changes to the document AST and apply only replacements to affected parts by SourceRange (Range<SourceLocation>) to the string (in its UTF-8 encoded form).
Beware that both column and line are 1-indexed! The start of the document is SourceLocation(line: 1, column: 1).
Now the line offset doesn’t help when you have the whole string in memory. Even with a C-string or rather String.UTF8View, we need absolute offsets, like String.UTF8View.Index. This involves translation of the ranges.
Soo …
- Get a UTF-8 view of your String;
- Iterate by lines like
cmarkdoes, where EOL (End of Line) is denoted bycmarkas either\nor\r(at least nothing weird happening here), so that … - … you map each
SourceRangeto aRange<String.Index>via itsString.UTV8View. (That is a lot of potentially costly index conversions.) - Replace the
Range<String.Index>in the original document with the replacement, aMarkdownFormatter-renderedStringof the replacement subtree.
In my case, I opted for a tiny Document as the holder for the original CodeBlock and the Paragraph with the Image inside. At least Document is a generic block container that can be turned into a string without trouble.
So this is a rather involved process for a task that sounded rather simple (given that swift-markdown looks so feature-packed on the surface).
In this my preliminary tests, the process works okay; here is a reformatted, annotated version of the code I’ve been trying today:
/// Collect these as you walk the document:
struct Delta {
let replacementRange: Markdown.SourceRange
let replacementDocument: Markdown.Document
}
/// Represent a Swift.String-applicable variant into its own type:
struct ApplicableDelta {
let delta: Delta
let utf8Range: Range<String.UTF8View.Index>
}
func isNewline(_ codeUnit: String.UTF8View.Element) -> Bool {
return codeUnit == String.UTF8View.Element(ascii: "\n")
|| codeUnit == String.UTF8View.Element(ascii: "\r")
}
let markdownString = ... // Load from file, provide inline
let utf8 = string.utf8
let lines = utf8.split(
omittingEmptySubsequences: false, // We need empty lines for proper line counting
whereSeparator: isNewline(_:)
)
func toUTF8(_ sourceLocation: SourceLocation) -> String.UTF8View.Index {
// Subtract -1 from line and column because in cmark, they are 1-indexed, not 0-indexed.
let lineIndex = lines[sourceLocation.line - 1].startIndex
return utf8.index(lineIndex, offsetBy: sourceLocation.column - 1)
}
let applicableDeltas: [ApplicableDelta] = deltas.map { delta in
let utf8LowerBound = toUTF8(delta.replacementRange.lowerBound)
let utf8UpperBound = toUTF8(delta.replacementRange.upperBound)
let utf8Range: Range<String.UTF8View.Index> = utf8LowerBound ..< utf8UpperBound
return ApplicableDelta(
delta: delta,
utf8Range: utf8Range
)
}
var newString = document.string
for applicableDelta
in applicableDeltas
.sorted(by: { $0.utf8Range.lowerBound < $1.utf8Range.lowerBound })
.reversed()
{
let renderedReplacementString = applicableDelta.delta.replacementDocument.format(options: .default)
// TODO: replace these two very costly O(n) range conversions
let stringLowerBound = applicableDelta.utf8Range.lowerBound.samePosition(in: newString)!
let stringUpperBound = applicableDelta.utf8Range.upperBound.samePosition(in: newString)!
let stringRange = stringLowerBound ..< stringUpperBound
newString.replaceSubrange(stringRange, with: renderedReplacementString)
}
// e.g. print(newString)
Check out the swift-markdown-babel repository in case you’re interested in per-node conversions of Markdown elements, and rendering the result as a diff.
The implementation of the last part, the UTF-8 offset conversion, is really, really work-in-progress – so much so that I’m almost ashamed to show it at all. But then I couldn’t have ended this post with a proper solution. So building in the open it is.
After all, to summarize, the swift-markdown API surface is really nice and the documentation is great.
When you do want to do anything more interesting than walk an existing document to e.g. collect all links, you need to change the package and maintain a fork or pick an existing one.
There are interesting changes in the wild, like enabling more CommonMark extensions (PR from 2021) or footnote support (PR from 2023). None of this can be added by extending existing objects via subclassing, for example, because that’s not how the API works.
All in all, this is a surprising limitation.
I didn’t expect to say that I miss the flexibility of subclassing NSObject; or in the worst case skip the Swift wrapper and go down a level to the actual C library to do the work I want to perform in my code after importing the package. But with swift-markdown, I can only either use what’s there, or need to fork the package itself.
This is a very inflexible approach to library building, taking away a lot of power. Usually, the trade-off is between power to one side, and ease of use and security on the other. But with this parser wrapper, I don’t see what locking-down and hiding the interesting internals actually buys us there.
I would prefer two major couple of changes to the package to open it up more:
- Change
MarkdownFormatterfromstructtoopen class. The value semantics protect the internal state of the formatter with each node visit, which sounds like a “Best Practice” on paper, but if all you can do with the formatter is runformat()once, it doesn’t really do anything. Subclassing would offer more interesting scenarios. - Publish all the things. The raw data is underscored, it’s not used a lot, but it’s necessary to declare your own types. At the moment, this makes tweaking the Markdown flavor you can work with impossible. As a remedy for the potential explosion in API surface, the formerly invisible stuff could be moved into a different package, so that most users will only import
Markdown, offering functionality just like now, andMarkdownInternalfor the hackers.
Not holding my breath, though :)