Clickable Swift Testing Failure Messages in Emacs compile-mode

God was this weird to figure out.

So swift test will produce a lot of stuff on standard output. Test failures, to nobody’s help, also just list the file names, not the (relative) paths:

􀢄  Test amazingTest() recorded an issue at MyTests.swift:12:34: Expectation failed: ...

In theory, MyTest.swift:12:34 is enough to instruct an editor where to look for the test failure.

Here’s a regular expression that matches this:

"^.* Test .* recorded an issue at ([a-zA-Z0-9_\/ ]+\.swift):([0-9]+):([0-9]+): "

Test it on regex101.com.

Now why do we need this – not just me, but you?

For Emacs compile-mode output, so that these lines are recognized as failures, and so that you can jump from one failure to the next with the help of next-error (C-x `), for example.

By default, a filename without a relative path won’t be found if you click the error messagt, though; resolved against the base project directory, the file cannot be found. A filename without a path is insanity. What if you have MyTests.swift in two test targets? Tough luck. (Spoiler: Even though we’ll get the filename resolution to work, I still have no clue how to deal with this, and I strongly believe that swift test should have a --print-path-in-output flag so we can rely on the proper source of truth, and not some band-aid.)

Let’s not get ahead of ourselves; how do we teach this format to the Emacs compilation mode error detector?

If you learn by example, read these:

If you hate yourself, read the manual (mirror). Oh wait, that’s compilation-error-regexp-alist-alist, where the regexp goes, not where it’s explained, read the other manual page on compilation-error-regexp-alist, where it explains:

Each elt has the form (REGEXP FILE [LINE COLUMN TYPE HYPERLINK HIGHLIGHT…]). If REGEXP matches, the FILE’th subexpression gives the file name, and the LINE’th subexpression gives the line number. The COLUMN’th subexpression gives the column number on that line.

If FILE, LINE or COLUMN are nil or that index didn’t match, that information is not present on the matched line. In that case the file name is assumed to be the same as the previous one in the buffer, line number defaults to 1 and column defaults to beginning of line’s indentation.

FILE can also have the form (FILE FORMAT…), where the FORMATs (e.g. “%s.c”) will be applied in turn to the recognized file name, until a file of that name is found. Or FILE can also be a function that returns (FILENAME) or (RELATIVE-FILENAME . DIRNAME). In the former case, FILENAME may be relative or absolute.

Let me help you decode that: “FILE’th” is shorthand for “if you provide a number, the number is the group index in the regexp”; with Swift, we have filename:line:column, so it’s in order, and we can pass 1 2 3 for the regexp printed above.

We can match this alright, but the path is not resolvable, so we need a function. To use a function, put the function name in place of the literal 1. The function will not take arguments.

How do you get the filename, in the function if it takes no arguments? You get it from the global regexp matching context: (match-string-no-properties 1) will behave like passing 1 to the alist. Or alist-alist.

For some reason of indirection, we have two alists:

  • compilation-error-regexp-alist: can contain the full alist, but actually nowadays only contains a symbol per language, the language’s name. Here: swift. The actual regexps are included in:
  • compilation-error-regexp-alist-alist, which is an alist of alists. Hence the name. We’ll add an entry in a minute so you see how is elements are structured.

First, here’s a working but probably not maximally efficient Swift path resolver based on a filename:

(defun ct/swift-compile-error-find-file ()
  "Find Swift test file from matched filename in compilation buffer.

Extracts filename from match group 1 and searches for it in Tests 
directories or .
Returns (FILENAME) or (RELATIVE-FILENAME . DIRNAME) for compilation mode."
  (let ((filename (match-string-no-properties 1)))
    (if (file-name-absolute-p filename)
        (list filename)
      (let* ((project-root (or (when (boundp 'compilation-directory)
                                 compilation-directory)
                               default-directory))
             (found-file (or
                          ;; First try exact path from project root
                          (when (file-exists-p (expand-file-name filename project-root))
                            (expand-file-name filename project-root))
                          ;; Then search in Tests directories
                          (car (file-expand-wildcards 
                                (expand-file-name (format "Tests/*/%s" filename) project-root)))
                          ;; Try deeper nesting
                          (car (file-expand-wildcards 
                                (expand-file-name (format "Tests/*/*/%s" filename) project-root)))
                          ;; Try Sources directories too (for non-test files referenced in tests, just in case)
                          (car (file-expand-wildcards 
                                (expand-file-name (format "Sources/*/%s" filename) project-root)))
                          (car (file-expand-wildcards 
                                (expand-file-name (format "Sources/*/*/%s" filename) project-root))))))
        (if found-file
            (list found-file)
          (list filename))))))

So this file-expand-wildcards usage will hopefully find all kinds of Swift files in Tests/. During interactive tests, I found that the glob will not match infinite subdirectories, so if you have deep directory structures – I’m sorry, then you need to add more deeply nested expansions. (Thanks again, swift test …)

So finally, we add the ‘swift’ symbol to both alists, and include the regular expression in weird Emacs Lisp form, where you need to escape the grouping parentheses:

(push 'swift compilation-error-regexp-alist)
(push '(swift 
        "^.* Test .* recorded an issue at \\([a-zA-Z0-9_/ ]+\\.swift\\):\\([0-9]+\\):\\([0-9]+\\): "
        ct/swift-compile-error-find-file 2 3)
      compilation-error-regexp-alist-alist)

Here you see that instead of just passing the group indices 1, 2, and 3, we pass the function name instead of the filename group index, and 2 and 3.

With this glorious change, you can click on test failures and use next-error and friends.

Now if you have the same test file name in two directories, you’re screwed. It’s ambiguous. Nothing to do about it. file-expand-wildcards will return a list of relative paths to the files anyway, which is very considerate, but it won’t help you decide which one is the correct one.

You could match the test case name in the regular expression, too, and then look for the test definition in both files and return the path of the file that contains the test case name. To further disambiguate, you could also use the line number to check for the location of the test case.

This sounds ridiculous because it is. We shouldn’t need to do this.

If you have two test files with two identical test case names at the exact same line number, you’re still screwed. I’m sorry.

Only thing that could help is a swift flag to include the relative path. Which would’ve been a great idea anyway. But alas.