Support for syntax test output highlighting

[deathaxe]

Problem description

Reading the syntax test output can sometimes be a bit tedious. A syntax highlighting of the output could propaply help improving the readability of the test output.

Preferred solution

To do so, the run_syntax_tests.py needs to be patched to add support for the syntax key in sublime-build files.

The proposal also contains a syntax definition. But I don't have a final idea about how to scope the different parts of the build output in detail. Maybe the scope names need to be tweaked a little bit.

1. Add support for the `syntax` parameter of `sublime-build` files to the `run_syntax_tests` command.

--- C:\Apps\Sublime Text 3\Data\Packages\Default\run_syntax_tests.py    Thu Oct 17 18:22:14 2019
+++ C:\Apps\Sublime Text 3\Data\Packages\Default\run_syntax_tests.py    Thu Oct 17 18:22:34 2019
@@ -10,7 +10,11 @@
 
 
 class RunSyntaxTestsCommand(sublime_plugin.WindowCommand):
-    def run(self, find_all=False, **kwargs):
+
+    def run(self,
+            find_all=False,
+            syntax="Syntax Test.sublime-syntax",
+            **kwargs):
 
         if not hasattr(self, 'output_view'):
             # Try not to call get_output_panel until the regexes are assigned
@@ -23,6 +27,7 @@
         settings.set('line_numbers', False)
         settings.set('gutter', False)
         settings.set('scroll_past_end', False)
+        settings.set('syntax', syntax)
 
         # Call create_output_panel a second time after assigning the above
         # settings, so that it'll be picked up as a result buffer

2. Add a Syntax Test.sublime-syntax file

%YAML 1.2
---
# http://www.sublimetext.com/docs/3/syntax.html
name: Sublime Syntax Tests
hidden: true
scope: text.syntax-tests
variables:
  scope_segment: '\w+(?:[\w-]*\+*)' # \+* is for the non standard scope.c++ scopes

contexts:
  main:
    - match: ^(\[)Finished(\])
      scope: markup.info.syntax-tests
      captures:
        1: punctuation.definition.markup.begin.syntax-tests
        2: punctuation.definition.markup.end.syntax-tests
    - match: ^(FAILED)(:) (\d+) of (\d+) assertions in (\d+) files failed
      captures:
        1: markup.error.failed.syntax-tests
        2: punctuation.separator.syntax-tests
        3: markup.error.num-failed.syntax-tests
        4: markup.info.num-asserts.syntax-tests
        5: markup.info.num-files.syntax-tests
    - match: ^
      push:
        - match: '[/.]'
          scope: punctuation.separator.path.syntax-tests
        - match: (:)(\d+)(:)(\d+)(:)
          captures:
            1: punctuation.separator.sequence.syntax-tests
            2: constant.numeric.integer.decimal.syntax-tests
            3: punctuation.separator.sequence.syntax-tests
            4: constant.numeric.integer.decimal.syntax-tests
            5: punctuation.separator.sequence.syntax-tests
          set:
            - match: \n
              pop: true
            - match: \[
              scope: punctuation.definition.selector.begin.syntax-tests
              push:
                - match: \]
                  scope: punctuation.definition.selector.end.syntax-tests
                  pop: true
                - match: '\)'
                  scope: invalid.illegal.stray-bracket.scope-selector
                - include: scope-selector

###[ PackageDev's Scope Selector ]#############################################

  scope-selector:
    - match: (?:^\s*)?({{scope_segment}})
      captures:
        1: string.unquoted.scope-segment.scope-selector
      push:
        - match: (\.)(\.+)?
          captures:
            1: punctuation.separator.scope-segments.scope-selector
            2: invalid.illegal.empty-scope-segment.scope-selector
          pop: true
        - match: \s+\(
          scope: invalid.illegal.operator-required-between-scope-and-group.scope-selector
        - match: \s+(?=\w)
          scope: keyword.operator.right.scope-selector
          pop: true
        - match: ''
          pop: true
    - match: '-'
      scope: keyword.operator.without.scope-selector
    - match: '&'
      scope: keyword.operator.with.scope-selector
    - match: '[,|]'
      scope: keyword.operator.or.scope-selector
    - match: \(
      scope: punctuation.section.group.begin.scope-selector
      push:
        - match: \)
          scope: punctuation.section.group.end.scope-selector
          set:
            - match: (\s*{{scope_segment}}|\.)+
              scope: invalid.illegal.operator-required-after-group.scope-selector
            - match: \s*
              pop: true
        - include: scope-selector

Alternatives

If the syntax definition seems too much, please, at least implement step 1 with Plain Text.tmLanguage as default syntax to enable custom sublime-build files with a custom syntax definition.

Maybe a 3rd party package like PackageDev could implement the highlighting then, without the need of patching the run_syntax_tests.py implementation or hack around to make it work.

[FichteFoll]

Even in the case 1. was implemented, the existing build system wouldn't know to use it and PD would have to provide a separate one leading to duplicate entries in the build system selection.

I think the build output of syntax tests is in general not optimized for human readability.

[deathaxe]

My current implementation looks like ...

Packages/Default/Syntax Tests.sublime-syntax

%YAML 1.2
---
# http://www.sublimetext.com/docs/3/syntax.html
name: Sublime Syntax Tests
hidden: true
scope: text.syntax-tests
variables:
  scope_segment: \w+(?:[\w-]*\+*) # \+* is for the non standard scope.c++ scopes

contexts:
  main:
    # summaries
    - match: ^(\[)Finished(\])
      scope: meta.summary.finished.syntax-tests markup.info.finished.syntax-tests
      captures:
        1: punctuation.definition.markup.begin.syntax-tests
        2: punctuation.definition.markup.end.syntax-tests
    - match: ^(FAILED)(:) (\d+) of (\d+) assertions in (\d+) files failed
      scope: meta.summary.failure.syntax-tests
      captures:
        1: markup.error.failed.syntax-tests
        2: punctuation.separator.syntax-tests
        3: constant.numeric.syntax-tests
        4: constant.numeric.syntax-tests
        5: constant.numeric.syntax-tests
    - match: ^(Success)(:) (\d+) assertions in (\d+) files passed
      scope: meta.summary.success.syntax-tests
      captures:
        1: markup.info.success.syntax-tests
        2: punctuation.separator.syntax-tests
        3: constant.numeric.syntax-tests
        4: constant.numeric.syntax-tests

    # locations
    - match: ^Packages/(..[^:]*)(:)(\d+)(:)(\d+)
      scope: meta.path.syntax-tests entity.name.section
      captures:
        2: punctuation.separator.sequence.syntax-tests
        3: constant.numeric.integer.decimal.syntax-tests
        4: punctuation.separator.sequence.syntax-tests
        5: constant.numeric.integer.decimal.syntax-tests

    # assertions
    - match: ^\s*(expect|found)(:)
      captures:
        1: keyword.other.syntax-tests
        2: punctuation.separator.syntax-tests
      push: assertion-body

  assertion-body:
    - match: \n
      pop: 1
    - match: '{{scope_segment}}'
      scope: string.unquoted.scope-segment.scope-selector
    - match: \.
      scope: punctuation.separator.scope-segments.scope-selector

Packages/Default/run_syntax_tests.py

--- run_syntax_tests.py	Sat Feb 10 16:02:06 2024
+++ run_syntax_tests.py	Sat Feb 10 16:02:36 2024
@@ -23,6 +23,7 @@
         settings.set('line_numbers', False)
         settings.set('gutter', False)
         settings.set('scroll_past_end', False)
+        settings.set("syntax", kwargs.get("syntax", "Packages/Default/Syntax Tests.sublime-syntax"))
 
         # Call create_output_panel a second time after assigning the above
         # settings, so that it'll be picked up as a result buffer
@@ -65,13 +66,15 @@
         total_assertions = 0
         failed_assertions = 0
 
+        pattern = re.compile(r'(.+):([^:]+):([^:]+): \[([^\]]+)\][^\[]+\[([^\]]+)\]')
+        repl = r'\1:\2:\3\n  expect: \4\n   found: \5\n'
+
         for t in tests:
             assertions, test_output_lines = sublime_api.run_syntax_test(t)
             total_assertions += assertions
             if len(test_output_lines) > 0:
                 failed_assertions += len(test_output_lines)
-                for line in test_output_lines:
-                    append(self.output_view, line + '\n')
+                append(self.output_view, ''.join(pattern.sub(repl, line) for line in test_output_lines))
 
         if failed_assertions > 0:
             message = 'FAILED: {} of {} assertions in {} files failed\n'

Resulting syntax test output looks like:

[BenjaminSchaaf]

Build 4170 adds the syntax parameter as requested. I have not added the sublime-syntax for this, it's not clear to me whether that would be best as an internal syntax definition or as part of the default packages.

[deathaxe]

Maybe treat it as regular expressions syntaxes, which are assigned to find inputs in Default but are provided via separate Regular Expressions package.

Note, the syntax as provided in my former comment however works only together with reformatted syntax test output, which is IMHO as important as syntax highlighting for 2 reasons:

1. long selectors cause output to soft-wrap on normal screens, which tanks readability and results in messed up wall of text, if many assertions fail.
2. expected and actual selector printed aligned on separate lines helps comparing them.

IMHO, reformatted display is even more important than syntax highlighting.

Maybe use symbols to reduce repetition of expect: and found: key words.

Long story short: We could add a Syntax Tests package (or any suitable alternative name) to sublimehq/Packages with a syntax definition for current test output.

I'd still keep going with my reformatted one as it eases up debugging though.

[deathaxe]

I slightly tweaked reformatted version to use symbols instead of text and update scopes.

It is part of https://github.com/deathaxe/sublime-commands

run_syntax_tests.py

--- Packages\Default\run_syntax_tests.py	Sat Feb 24 16:28:59 2024
+++ Packages\Default\run_syntax_tests.py	Sat Feb 24 16:29:05 2024
@@ -75,18 +68,15 @@
         total_assertions = 0
         failed_assertions = 0
 
-        pattern = re.compile(r"(.+):([^:]+):([^:]+): \[([^\]]+)\][^\[]+\[([^\]]+)\]")
-        repl = r"\1:\2:\3\n  ✓ \4\n  ! \5\n"
+        pattern = re.compile(r'(.+):([^:]+):([^:]+): \[([^\]]+)\][^\[]+\[([^\]]+)\]')
+        repl = r'\1:\2:\3\n  ✓ \4\n  ! \5\n'
 
         for t in tests:
             assertions, test_output_lines = sublime_api.run_syntax_test(t)
             total_assertions += assertions
             if len(test_output_lines) > 0:
                 failed_assertions += len(test_output_lines)
-                append(
-                    self.output_view,
-                    "".join(pattern.sub(repl, line) for line in test_output_lines),
-                )
+                append(self.output_view, ''.join(pattern.sub(repl, line) for line in test_output_lines))
 
         if failed_assertions > 0:
             message = "FAILED: {} of {} assertions in {} files failed\n"

Sublime Syntax Test Results.sublime-syntax

%YAML 1.2
---
# https://www.sublimetext.com/docs/syntax.html
name: Sublime Syntax Test Results
hidden: true
scope: text.sublime.syntax-test-results
variables:
  scope_segment: \w+(?:[\w-]*\+*) # \+* is for the non standard scope.c++ scopes

contexts:
  main:
    # comments
    - match: ^\#.+\n?
      scope: comment.line.double-dash.syntax-test-results
    # summaries
    - match: ^(\[)Finished(\])
      scope: meta.summary.finished.sublime.syntax-test-results markup.info.finished.sublime.syntax-test-results
      captures:
        1: punctuation.definition.markup.begin.sublime.syntax-test-results
        2: punctuation.definition.markup.end.sublime.syntax-test-results
    - match: ^(FAILED)(:) (\d+) of (\d+) assertions in (\d+) files failed
      scope: meta.summary.failure.sublime.syntax-test-results
      captures:
        1: markup.error.failed.sublime.syntax-test-results
        2: punctuation.separator.other.sublime.syntax-test-results
        3: meta.assertions.failed.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
        4: meta.assertions.count.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
        5: meta.test-files.count.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
    - match: ^(Success)(:) (\d+) assertions in (\d+) files passed
      scope: meta.summary.success.sublime.syntax-test-results
      captures:
        1: markup.info.success.sublime.syntax-test-results
        2: punctuation.separator.other.sublime.syntax-test-results
        3: meta.assertions.count.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
        4: meta.test-files.count.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
    # locations
    - match: ^(?=Packages/)
      push: location
    - match: ^\s*(✓)\s
      captures:
        1: markup.info.sublime.syntax-test-results
      push: expected-scope
    - match: ^\s*(!)\s
      captures:
        1: markup.error.sublime.syntax-test-results
      push: found-scope

  location:
    - meta_content_scope: meta.path.filename.sublime.syntax-test-results entity.name.filename.sublime.syntax-test-results
    - match: (:)(\d+)(:)(\d+)
      captures:
        1: meta.path.row.sublime.syntax-test-results punctuation.separator.row.sublime.syntax-test-results
        2: meta.path.row.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
        3: meta.path.column.sublime.syntax-test-results punctuation.separator.column.sublime.syntax-test-results
        4: meta.path.column.sublime.syntax-test-results meta.number.integer.decimal.sublime.syntax-test-results constant.numeric.value.sublime.syntax-test-results
      pop: 1
    - match: /
      scope: punctuation.separator.path.sublime.syntax-test-results
    - match: $
      pop: 1

  expected-scope:
    - meta_content_scope: meta.scope.expected.sublime.syntax-test-results
    - match: \n
      pop: 1
    - include: scope-selectors

  found-scope:
    - meta_content_scope: meta.scope.found.sublime.syntax-test-results
    - match: \n
      pop: 1
    - include: scope-selectors

  scope-selectors:
    - meta_scope: meta.group.scope-selector
    - match: \(
      scope: punctuation.section.group.begin.scope-selector
      push: scope-selectors
    - match: \)
      scope: punctuation.section.group.end.scope-selector
      pop: 1
    - match: '{{scope_segment}}'
      scope: string.unquoted.scope-segment.scope-selector
    - match: \.
      scope: punctuation.separator.scope-segments.scope-selector
    - match: '-'
      scope: keyword.operator.without.scope-selector
    - match: '&'
      scope: keyword.operator.with.scope-selector
    - match: '[,|]'
      scope: keyword.operator.or.scope-selector

Mariana displays it as

With some additional color scheme rules added to my custom Brackets Dark color scheme it looks like

[michaelblyons]

If you're going to change the output formatting, IMO why not also roll up into a single error where expected/actual are also the same for consecutive characters on the same line?

Packages/C#/tests/syntax_test_C#11.cs:26:5-12

Edit: Maybe that's harder. Sorry.

[deathaxe]

Not sure if next_result/prev_result would pick those resulting regions up.

[mataha]

If it's not possible to decouple the default test result syntax from the package, could it perhaps be inlined given its simplicity? I'm aware that syntax accepts filenames only - maybe a helper function or a separate setting?

Sample code

diff --git a/run_syntax_tests.py b/run_syntax_tests.py
index 8ec24b73b4a2..b05d53f80f4b 100644
--- a/run_syntax_tests.py
+++ b/run_syntax_tests.py
@@ -7,12 +7,17 @@ import sublime_plugin


 PACKAGES_FILE_REGEX = r'^Packages/(..[^:]*):([0-9]+):?([0-9]+)?:? (.*)$'
+DEFAULT_OUTPUT_VIEW_SYNTAX = '''
+
+...
+
+'''


 class RunSyntaxTestsCommand(sublime_plugin.WindowCommand):
     def run(self,
             find_all=False,
-            syntax='Plain text.tmLanguage',
+            syntax=None,
             **kwargs):

         if not hasattr(self, 'output_view'):
@@ -26,7 +31,11 @@ class RunSyntaxTestsCommand(sublime_plugin.WindowCommand):
         settings.set('line_numbers', False)
         settings.set('gutter', False)
         settings.set('scroll_past_end', False)
-        settings.set('syntax', syntax)
+        if syntax is None:
+            # Should create an in-memory syntax?
+            settings.set('syntax', DEFAULT_OUTPUT_VIEW_SYNTAX)
+        else:
+            settings.set('syntax', syntax)

         # Call create_output_panel a second time after assigning the above
         # settings, so that it'll be picked up as a result buffer

Loosely related: soft-wrapping is indeed a pain point:

[FichteFoll]

What would be the benefit of a "package-decoupled" syntax definition? Having it as a resource (a file in a package) that you can reference works fine and would be preferred in most situations, imo. Unless you intend to do dynamic changes to the definition, in which we'd need to talk about this specific use case in more detail because there is likely already a satisfactory solution available or it requires larger architectural changes for the syntax and highlighting system.

[mataha]

Not my intention - just wanted to point out the fact that there is no consensus where that definition should live (default packages vs. own package) when, in my opinion, a syntax as strongly coupled with a plugin as this one should be present as close to it as possible.

[deathaxe]

If you're going to change the output formatting, IMO why not also roll up into a single error where expected/actual are also the same for consecutive characters on the same line?

Packages/C#/tests/syntax_test_C#11.cs:26:5-12

Edit: Maybe that's harder. Sorry.

You are right.

Implemented at https://github.com/deathaxe/sublime-commands/blob/master/run_syntax_tests.py

It wasn't too hard after all.

After some experimenting deduplicating results actually reduces noise and improves navigation speed - less key strokes needed to move on.

[BenjaminSchaaf]

I don't think it make sense to have the syntax for test output anywhere but the default package, ie. in core. I've taken the opportunity to rework the test errors. Subject to change of course here's what they will look like:

Packages/Batch File/tests/syntax_test_batch_file.bat:2035:14
error: test covers non-existing text
2034 | put arg1 arg2
2035 | :: ^^^^^^^^^^^^^ - meta.function-call meta.function-call
     |               ^^ these characters don't exist

Packages/C++/syntax_test_c.c:505:12
error: scope does not match
505 | struct point **alloc_points();
508 | /*          ^^^^ keyword.operator */
    |             -  - these locations did not match
actual:
    |             ^               source.c 
    |              ^              source.c keyword.operator.c 
    |               ^             source.c keyword.operator.c 
    |                ^^^^^^^^^^^^ source.c meta.function.c entity.name.function.c 

Packages/C++/syntax_test_c.c:513:5
error: unknown symbol specifier `a`
513 | /*@@ a*/
    |      ^ expected one of `none`, `local-definition`, `global-definition`, `definition` or `reference`

Packages/C++/syntax_test_c.c:731:6
error: unexpected symbol `return_type_pointer_no_space` (definition) found
note: all symbols must be matched unless `partial-symbols` is specified
731 | int* return_type_pointer_no_space(){}
    |      @@@@@@@@@@@@@@@@@@@@@@@@@@@@ definition

This is using the following syntax definition:

%YAML 1.2
---
# http://www.sublimetext.com/docs/syntax.html
name: Syntax Tests
hidden: true
scope: text.syntax-tests
contexts:
  main:
    - match: '^([^:\n]*[/\\]syntax_test_[^:\n]*)((:)([0-9]+))?((:)([0-9]+))?\n'
      captures:
        1: entity.name.filename.syntax-tests
        3: punctuation.separator.sequence.syntax-tests
        4: constant.numeric.syntax-tests
        6: punctuation.separator.sequence.syntax-tests
        7: constant.numeric.syntax-tests
    - match: '^error: .*'
      scope: message.error.syntax-tests
    - match: '^note: .*'
      scope: comment.line.syntax-tests
    - match: '^ *([0-9]+) (\|) (.*)'
      captures:
        1: constant.numeric.syntax-tests
        2: punctuation.separator.sequence.syntax-tests
    - match: '^ *(\|)([ ^@-]+)(.*)'
      captures:
        1: punctuation.separator.sequence.syntax-tests
        2: punctuation.definition.syntax-tests
        3: comment.line.syntax-tests
    - match: ^(actual)(:)
      captures:
        1: keyword.other.syntax-tests
    - match: ^(FAILED)(:) (\d+) of (\d+) assertions in (\d+) files failed
      scope: meta.summary.failure.syntax-tests
      captures:
        1: markup.error.failed.syntax-tests
        2: punctuation.separator.syntax-tests
        3: constant.numeric.syntax-tests
        4: constant.numeric.syntax-tests
        5: constant.numeric.syntax-tests
    - match: ^(Success)(:) (\d+) assertions in (\d+) files passed
      scope: meta.summary.success.syntax-tests
      captures:
        1: markup.info.success.syntax-tests
        2: punctuation.separator.syntax-tests
        3: constant.numeric.syntax-tests
        4: constant.numeric.syntax-tests

Here's a screenshot of mariana:

I'd like to get feedback on this - particularly in regards to the syntax - before going forward.

[michaelblyons]

The split "actual" for keyword.operator.c: is that from tokenization?

[BenjaminSchaaf]

@michaelblyons yep

[deathaxe]

I like the new representation as it gives detailed and grouped information about test failures.

Proper feedback on syntax definitions requires some time for evaluation.

So let's start with a structural question.

Syntax tests appear to follow a schema of

1. error message / note
2. failed test line
3. expected scope
4. mismatch message
5. actual scopes

So instead of...

we could use that information to scope 3. comment by pushing a context on stack.

How about symbol tests' structure. They seem to miss the printed comment from syntax test file, or is it just the row number which is missing in the pseudo gutter?

Packages/C++/syntax_test_c.c:731:6
error: unexpected symbol `return_type_pointer_no_space` (definition) found
note: all symbols must be matched unless `partial-symbols` is specified
731 | int* return_type_pointer_no_space(){}
    |      @@@@@@@@@@@@@@@@@@@@@@@@@@@@ definition

I would expect something like

Packages/C++/syntax_test_c.c:731:6
error: unexpected symbol `return_type_pointer_no_space` (definition) found
note: all symbols must be matched unless `partial-symbols` is specified
731 | int* return_type_pointer_no_space(){}
732 | /*      @@@@@@@@@@@@@@@@@@@@@@@@@ definition */
    |      ^^^ unmatched symbol

EDIT: Or is by chance the last line @@@@ ... definition already the failure message (4.)? and (3.) is just not present as it hasn't been defined in syntax test file?

[michaelblyons]

Can you show what it looks like when two assertion lines fail for the same line of code?

[michaelblyons]

It's not super important, but I think the - in the non-matching locations should still use ^.

Also, if you use DeathAxe's pushed context, you could add keyword.operator.logical to -, ,, and | in the assertions.

[deathaxe]

The question is how much complexity is acceptable. The initial suggestion looks rather sublime aka. minimalistic.

[deathaxe]

Suggestions

General scope suggestions

1. I'd suggest to apply detailed unique meta scopes to enable color schemes to target everything.
2. Always include \n in patterns, so backgrounds can be applied to full lines.
3. It is probably more appropriate to avoid keyword scopes in text documents and use something like markup instead.

Scope assertion violation location

1. Scoping whole line (e.g.: meta.path) would enable color schemes to tint whole line's background with a single simple selector like text.syntax-tests meta.path

2. Following Find Results syntax, scope line- and column numbers constant.numeric.[line-number|column-number]

    - match: ^([^:\n]*[/\\]syntax_test_[^:\n]*)(?:(:)([0-9]+))?(?:(:)([0-9]+))?(\n)
      scope: meta.path.syntax-tests                      # <- new
      captures:
        1: entity.name.filename.syntax-tests
        2: punctuation.separator.sequence.syntax-tests
        3: constant.numeric.line-number.syntax-tests     # <- modified
        4: punctuation.separator.sequence.syntax-tests
        5: constant.numeric.column-number.syntax-tests   # <- modified
        6: meta.fold.begin.syntax-tests                  # <- new (see next section)

Note: Unused capture groups have been removed from pattern.

Apply syntax based folding

Indentation folding causes unexpected behavior.

It's rather desirable to be able to consistently fold all additional debug information and keep only assertion violation locations (file paths) visible.

Therefore the \n of each assertion location and the empty line after the debug information are scoped meta.fold.[begin|end], which is already in use by some other syntax definitions.

  - match: ^\n
    scope: meta.fold.end.syntax-tests  

Alternatively punctuation.section.block.[begin|end] may be appropriate choices, if \n is accepted to be treated punctuation.

Folding rules via e.g. Fold Syntax Test Results.tmPreferences would be:

<?xml version="1.0" encoding="UTF-8"?>
<plist version="1.0">
<dict>
    <key>scope</key>
    <string>text.syntax-tests</string>
    <key>settings</key>
    <dict>
        <key>indentationFoldingEnabled</key>
        <false/>
        <key>foldScopes</key>
        <array>
            <dict>
                <key>begin</key>
                <string>meta.fold.begin</string>
                <key>end</key>
                <string>meta.fold.end</string>
                <key>excludeTrailingNewlines</key>
                <false/>
            </dict>
        </array>
    </dict>
</dict>
</plist>

Message Scopes

1. It's the first time I come accross top-level scope message which is used to scope error: ....

     - match: '^error: .*'
       scope: message.error.syntax-tests
   

   It seems however already being used by various output syntaxes, such as Cargo Output DMD Output and Make output

   A commonly used scope pattern used by linters is markup.[error|warning|info].

   Would it therefore make sense to scope note: message.info instead of comment.line?

     - match: '^note: .*'
       scope: message.info.syntax-tests
   

   It requires a new rule for Mariana and maybe Monokai to provide desired default highlighting.

   {
      "rules": [
        {
          "name": "Build Output Error Message",
          "scope": "message.error",
          "foreground": "var(red)"
        },
        {
          "name": "Build Output Warning Message",
          "scope": "message.warning",
          "foreground": "var(orange)"
        },
        {
          "name": "Build Output Info Message",
          "scope": "message.info",
          "foreground": "var(green)"
        },
      ],
   }
   

2. It would probably make sense to scope leading keyword, so color schemas can target it and apply distinct color or font style.

       - match: ^(error(:)) .*\n
         scope: message.error.syntax-tests
         captures:
           1: markup.heading.syntax-tests
           2: punctuation.separator.syntax-tests
       - match: ^(note(:)) .*\n
         scope: message.info.syntax-tests
         captures:
           1: markup.heading.syntax-tests
           2: punctuation.separator.syntax-tests
   

   If default bold font style of markup.heading is not desired, corresponding color scheme rules to style them e.g. italic would be:

   {
      "rules": [
        {
          "name": "Build Output Error Message Heading",
          "scope": "message.error markup.heading",
          "font_style": "italic",
        },
        {
          "name": "Build Output Warning Message Heading",
          "scope": "message.warning markup.heading",
          "font_style": "italic",
        },
        {
          "name": "Build Output Info Message Heading",
          "scope": "message.info markup.heading",
          "font_style": "italic",
        },
      ],
   }
   

Gutter Scopes

It probably may make sense to target the pseudo gutter and line numbers.

    - match: ^( *([0-9]+) (\|)) (.*)
      captures:
        1: meta.gutter.syntax-tests           # <-- new
        2: constant.numeric.line-number.syntax-tests
        3: punctuation.separator.gutter.syntax-tests
    - match: ^( *(\|)) ([ ^@-]+)(.*)
      captures:
        1: meta.gutter.syntax-tests           # <-- new
        2: punctuation.separator.gutter.syntax-tests
        3: punctuation.definition.syntax-tests
        4: comment.line.syntax-tests

2034 | put arg1 arg2
^^^^^^ meta.gutter
^^^^ constant.numeric.line-number
     ^ punctuation.separator.gutter

Color Schemes can then target gutter background, line numbers and separators.

{
  "rules": [
    {
      "name": "Syntax Test Results: Gutter Background",
      "scope": "text.syntax-tests meta.gutter",
      "background": "color(var(blue4) alpha(0.08))",
    },
    {
      "name": "Syntax Test Results: Gutter Separator",
      "scope": "text.syntax-tests meta.gutter punctuation.separator",
      "foreground": "var(blue4)",
    },
    {
      "name": "Syntax Test Results: Gutter Line Numbers",
      "scope": "text.syntax-tests meta.gutter constant.numeric",
      "foreground": "var(blue6)",
    },
  ],
}

Summary Scopes

Summary line could/should follow normal message scoping.

Hence using

• message.[error|info]
• markup.heading

  - match: ^(FAILED(:)) (\d+) of (\d+) assertions in (\d+) files failed\n?
    scope: meta.summary.failure.syntax-tests message.error.syntax-tests
    captures:
      1: markup.heading.syntax-tests
      2: punctuation.separator.syntax-tests
      3: constant.numeric.syntax-tests
      4: constant.numeric.syntax-tests
      5: constant.numeric.syntax-tests
  - match: ^(Success(:)) (\d+) assertions in (\d+) files passed\n?
    scope: meta.summary.success.syntax-tests message.info.syntax-tests
    captures:
      1: markup.heading.syntax-tests
      2: punctuation.separator.syntax-tests
      3: constant.numeric.syntax-tests
      4: constant.numeric.syntax-tests

Performance

It turns out syntax parsing performance can be improved by about 15-17% by pushing a dedicated violation-details context on stack, which contains the patterns for debug messages.

Additional Info

Syntax Test Results.sublime-syntax

%YAML 1.2
---
# http://www.sublimetext.com/docs/syntax.html
name: Syntax Test Results
scope: text.syntax-tests
contexts:
  main:
    # Assertion Violation Location
    - match: ^([^:\n]*[/\\]syntax_test_[^:\n]*)(?:(:)([0-9]+))?(?:(:)([0-9]+))?(\n)
      scope: meta.path.syntax-tests
      captures:
        1: entity.name.filename.syntax-tests
        2: punctuation.separator.sequence.syntax-tests
        3: constant.numeric.line-number.syntax-tests
        4: punctuation.separator.sequence.syntax-tests
        5: constant.numeric.column-number.syntax-tests
        6: meta.fold.begin.syntax-tests
      # note: parsing details in separate context increases performance by about 17%
      push: violation-details

    # Summary
    - match: ^(FAILED(:)) (\d+) of (\d+) assertions in (\d+) files failed\n?
      scope: meta.summary.failure.syntax-tests message.error.syntax-tests
      captures:
        1: markup.heading.syntax-tests
        2: punctuation.separator.syntax-tests
        3: constant.numeric.syntax-tests
        4: constant.numeric.syntax-tests
        5: constant.numeric.syntax-tests
    - match: ^(Success(:)) (\d+) assertions in (\d+) files passed\n?
      scope: meta.summary.success.syntax-tests message.info.syntax-tests
      captures:
        1: markup.heading.syntax-tests
        2: punctuation.separator.syntax-tests
        3: constant.numeric.syntax-tests
        4: constant.numeric.syntax-tests

  violation-details:
    # Messages
    - match: ^(error(:)) .*\n
      scope: message.error.syntax-tests
      captures:
        1: markup.heading.syntax-tests
        2: punctuation.separator.syntax-tests
    - match: ^(note(:)) .*\n
      scope: message.info.syntax-tests
      captures:
        1: markup.heading.syntax-tests
        2: punctuation.separator.syntax-tests

    # Assertion Line
    - match: ^( *([0-9]+) (\|)) .*
      captures:
        1: meta.gutter.syntax-tests
        2: constant.numeric.line-number.syntax-tests
        3: punctuation.separator.gutter.syntax-tests
      push: expected-scopes

    # Mismatch Message
    - match: ^( *(\|)) ([ ^@-]+)(.*)
      captures:
        1: meta.gutter.syntax-tests
        2: punctuation.separator.gutter.syntax-tests
        3: punctuation.definition.syntax-tests
        4: comment.line.syntax-tests

    # Actual Scopes Block
    - match: ^(actual(:))\n
      scope: meta.separator.syntax-tests
      captures:
        1: markup.heading.syntax-tests
        2: punctuation.separator.syntax-tests
      push: actual-scopes

    - match: ^\n?
      scope: meta.fold.end.syntax-tests
      pop: 1

  expected-scopes:
    - match: ^( *([0-9]+) (\|)) (.*)
      captures:
        1: meta.gutter.syntax-tests
        2: constant.numeric.line-number.syntax-tests
        3: punctuation.separator.gutter.syntax-tests
        4: meta.scope.expected.syntax-tests comment.line.syntax-tests
    - match: ^
      pop: 1

  actual-scopes:
    - match: ^( *(\|))([ ^@-]+)(.*)
      captures:
        1: meta.gutter.syntax-tests
        2: punctuation.separator.gutter.syntax-tests
        3: punctuation.definition.syntax-tests
        4: meta.scope.actual.syntax-tests comment.line.syntax-tests
    - match: ^
      pop: 1

Additional Color Scheme Rules for illustration of how color schemes may want to address background of certain scopes and why it is therefore important to include newline.

Mariana.sublime-color-scheme

{
  "rules": [

    /*
     * Syntax Test Results
     */

    {
      "name": "Syntax Test Results: Assertion Violation Location",
      "scope": "text.syntax-tests meta.path",
      "background": "color(var(blue4) alpha(0.2))",
    },
    {
      "name": "Syntax Test Results: Common Background",
      "scope": "text.syntax-tests message, text.syntax-tests meta.separator, text.syntax-tests meta.fold.end",
      "background": "color(var(blue4) alpha(0.08))",
    },
    {
      "name": "Syntax Test Results: Gutter Background",
      "scope": "text.syntax-tests meta.gutter",
      "background": "color(var(blue4) alpha(0.08))",
    },
    {
      "name": "Syntax Test Results: Gutter Separator",
      "scope": "text.syntax-tests meta.gutter punctuation.separator",
      "foreground": "var(blue4)",
    },
    {
      "name": "Syntax Test Results: Gutter Line Numbers",
      "scope": "text.syntax-tests meta.gutter constant.numeric",
      "foreground": "var(blue6)",
    },
    {
      "name": "Syntax Test Results: Actual Scope Heading",
      "scope": "text.syntax-tests meta.separator markup.heading",
      "font_style": "italic",
    },

    /*
     * Build Output Messages
     */

    {
      "name": "Build Output Error Message",
      "scope": "message.error",
      "foreground": "var(red)"
    },
    {
      "name": "Build Output Error Message Heading",
      "scope": "message.error markup.heading",
      "font_style": "italic",
    },
    {
      "name": "Build Output Warning Message",
      "scope": "message.warning",
      "foreground": "var(orange)"
    },
    {
      "name": "Build Output Warning Message Heading",
      "scope": "message.warning markup.heading",
      "font_style": "italic",
    },
    {
      "name": "Build Output Info Message",
      "scope": "message.info",
      "foreground": "var(green)"
    },
    {
      "name": "Build Output Info Message Heading",
      "scope": "message.info markup.heading",
      "font_style": "italic",
    },
  ],
}  

[keith-hall]

error: test covers non-existing text

This is going to become an error now? There are some cases where it is convenient to test the start of the next line using "overspill" due to how the syntax test assertions can interfere with syntax highlighting i.e. the observer effect

[BenjaminSchaaf]

@deathaxe I've taken most of those suggestions on board, thanks for putting that syntax together and so quickly!

@keith-hall Tests wrapping to the next line leads to very odd looking error messages, but since this is a relied upon feature I've reverted adding that error. You'll just need to deal with the odd-looking error messages.

[jwortmann]

If the - character is used to mark the non-matching locations, could it get a unique scope, so that we can highlight it in bright red in our color schemes? Currently all of the ^@- markers seem to be scoped punctuation.definition.syntax-tests, but I'd like to at least not mix the error locations with the ^ markers for the actual scope names, if possible.

    |             -  - these locations did not match
actual:
    |             ^               source.c 
    |              ^              source.c keyword.operator.c 
    |               ^             source.c keyword.operator.c 
    |                ^^^^^^^^^^^^ source.c meta.function.c entity.name.function.c 

[deathaxe]

I haven't touched selectors' scopes as anything else than comment caused the output to feel noisy and overwhelming. Unique scopes may however be useful.

I was satisfied with tinting punctuation.definition a bit more light, even without any distinction between mismatch line and acutal scope indicators.

Keeping [^@-] together is probably required as mismatching symbols are printed using @ as it seems (see @@@...@ definition line).

Tinting those in red is not ideal for mismatching symbols.

[michaelblyons]

Can you show what it looks like when two assertion lines fail for the same line of code?

Bump

[BenjaminSchaaf]

@michaelblyons They just show up as separate errors.

[BenjaminSchaaf]

The new error messages have shipped with build 4181.