TextMate scope selectors: scope exclusion is not implemented

[aeschli]

From @monk-time on September 4, 2017 14:51

• VSCode Version: Code 1.15.1 (41abd21afdf7424c89319ee7cb0445cc6f376959, 2017-08-16T18:07:25.676Z)
• OS Version: Windows_NT x64 6.1.7601
• Extensions:

Extension	Author (truncated)	Version
intellij-idea-keybindings	k--	0.2.13
selectline-statusbar	tom	0.0.2

Also the same result with today's VS Code Insiders build.

---

According to the documentation for TextMate scope selectors, VSCode supports the syntax for excluding matching scopes:

entity.name.method - source.java matches all scopes that start with entity.name.method but not if a parent scope matches source.java

This functionality is used in at least one built-in theme that I could find: https://github.com/Microsoft/vscode/blob/c00bdb74ee665cccfc5c4e41520893bb19ef61e5/extensions/theme-monokai/themes/monokai-color-theme.json#L316

But it seems that this syntax makes the selector with - in it invalid, and VSCode doesn't apply the given rule to anything at all.

Steps to Reproduce:

1. Put this rule for syntax highlighting in the settings. Confirm that comments turn yellow at least in some files.

{
    "editor.tokenColorCustomizations": {
        "textMateRules": [
            {
                "scope": "comment",
                "settings": {
                    "foreground": "#FFFF00"
                }
            }
        ]
    }
}

2. Change the line with the scope selector to: "scope": "comment - source.js", Expected result: comments that changed their color after step 1 remain yellow in all files except in .js. Actual result: all comments reset to a color defined by the current theme.

Reproduces without extensions: Yes

Copied from original issue: Microsoft/vscode#33802

[curiouslychase]

just to make sure it's captured, the descendant scope selectors such as "source.ts meta.import-equals.external.ts" aren't working either (mentioned from Microsoft/vscode#34909), so it seems like anything beyond a single selector isn't being applied.

[chbk]

This is really lacking. Also, scope exclusion should not be limited to parent scopes but should be similar to the CSS :not() selector. To highlight string but not string.docstring for example.

[Mooninaut]

As an example of the inconvenience not having descendant selectors causes, in my personal theme, instead of the scope ["keyword source.python", "punctuation source.python"], I'm using this quasi-exhaustive list, which I keep having to add to:

[ "keyword.operator.arithmetic.python", "keyword.operator.assignment.python", "keyword.operator.comparison.python", "punctuation.definition.arguments.begin.python", "punctuation.definition.arguments.end.python", "punctuation.definition.dict.begin.python", "punctuation.definition.dict.end.python", "punctuation.definition.list.begin.python", "punctuation.definition.list.end.python", "punctuation.section.function.lambda.begin.python", "punctuation.separator.annotation.result.python", "punctuation.separator.arguments.python", "punctuation.separator.colon.python", "punctuation.separator.element.python", "punctuation.separator.period.python" ]

[frou]

Here's an example of a selector expression using exclusion (and excluded exclusion!) in a Sublime Text theme. It's very nice to have the power!

[CodingMarkus]

Only the main scope is applied. E.g. keyword - meta.preprocessor behaves exactly like keyword - meta. I assume that comment - source.js in fact behaves like comment - source. So it's not correct that exclusion is not implemented at all, it's just not implemented correctly.

[jaymegordo]

+1 for exclusions please!

[lilyball]

Only the main scope is applied. E.g. keyword - meta.preprocessor behaves exactly like keyword - meta. I assume that comment - source.js in fact behaves like comment - source. So it's not correct that exclusion is not implemented at all, it's just not implemented correctly.

Are you sure about that? I just tried using a scope exclusion in editor.tokenColorCustomizations and it appears to have simply disabled the rule (I'm not sure if there's anywhere to look for logging on this). I even tried writing it as just markup - meta and it still affected nothing at all (and I verified in my test document that there was plenty of text that matched markup without matching meta).

[ElectricRCAircraftGuy]

In my testing, it appears I can get effects similar to "exclusions" simply by specifying a very narrow scope. Ex:

"scope": "comment.block.documentation.cpp variable.parameter.cpp"

Full context:

            "textMateRules": [
                {
                    "name": "Doxygen variable names",
                    // This works! YOU **MUST** put the wider scope FIRST, and then the narrower scope after!
                    "scope": [
                        "comment.block.documentation.cpp variable.parameter.cpp",
                        "comment.line.double-slash.documentation.cpp variable.parameter.cpp",
                    ],
                    "settings": {
                        "foreground": "#a6a292",
                    },
                },
                {
                    "name": "Doxygen keywords, such as 'param' and 'brief'",
                    "scope": "storage.type.class.doxygen.cpp",
                    "settings": {
                        "foreground": "#7C8E9C",
                    }
                },
            ]

See this in my tutorial here, including screenshots and explanations: How to customize colors in VSCode by using the built-in "Scope inspector" tool.

[lilyball]

It appears that TextMate also had a bunch of other operators not documented in the manual. Sublime Text also supports these This is detailed in this StackOverflow answer and links to a bunch of blog posts and mailing list posts by Alan Odgaard explaining them. It also lists other operators Sublime Text doesn't support, though my suspicion is these operators were added in TextMate 2.0.

It would be really helpful if VSCode could support all of the TextMate 1.x operators, i.e. the operators Sublime Text supports, both because they're really useful and because they're found in existing Sublime Text grammars.

Personally, I would find the > operator (which SublimeText doesn't support) to be extremely useful. It would solve the problem I'm having right now in my editor.tokenColorCustomizations (trying to color string.interpolated > punctuation.definition without affecting other string punctuation and properly handling arbitrary nesting within strings).

[lilyball]

Looking at the actual source of this repo, it looks like it's actually trying to support L:, R:, -, (…), , and /| (though it's still missing & for intersection).

Given that, I'm not sure what's going on here. Even trivial grouping doesn't work in my experiments (in editor.tokenColorCustomizations). "scope": "string.interpolated" works but "scope": "(string.interpolated)" doesn't.

[ghost]

• https://github.com/atom/first-mate/blob/v7.4.2/src/scope-selector-matchers.coffee
• https://github.com/atom/first-mate/blob/v7.4.2/src/scope-selector-parser.pegjs
• https://github.com/atom/first-mate/blob/v7.4.2/src/scope-selector.coffee

This is how Atom implemented it.

[ghost]

- - scope is explicitly inalid syntax in the Atom implementation, best to remove it for upstream compat.

~~Also - scope was incredibly problematic because of there being no LHS selector so those need removing too.~~

[ghost]

Good and bad news - I implemented the first-mate selector scripts as pure JS. However there are multiple regressions and loss of versatility as Microsoft is not rigidly adhering to the Textmate selector specification right now:

• Scopes are matched positionally and literally in Atom - meta.parens does not match meta.function-call.parens.
• Negation doesn't appear to work recursively in Atom the way it's meant to here.

Here's the files:

• https://github.com/SNDST00M/vscode-textmate-languageservice/blob/v0.2.0/src/util/scope-selector-parser.pegjs
• https://github.com/SNDST00M/vscode-textmate-languageservice/blob/v0.2.0/src/util/scope-selector-parser.js
• https://github.com/SNDST00M/vscode-textmate-languageservice/blob/v0.2.0/src/util/scope-selector-matchers.js
• https://github.com/SNDST00M/vscode-textmate-languageservice/blob/v0.2.0/src/util/scope-selector.ts

The parser is not Typescript and I'd appreciate help making it type-sensitive!

[ghost]

@kevinsawicki @GlenCFL @50Wliu are any of you able to assist in translating your previous work to TypeScript, mainly the scope-selector-matchers.js file that I've linked in the comment above?

[50Wliu]

Sorry - I don't have much context here. Is the issue that VS Code doesn't support exclusions or "spaces", but Atom does? What is the end goal of converting Atom's scope selector files, and how does that fit into vscode-textmate?

[ghost]

1. Yes, Atom's Textmate files support a wide variety of syntax and are spec-ready.
2. The goal is to get them both running on the same Textmate specification.
3. vscode-textmate is written in TypeScript instead of CoffeeScript and it'd be beneficial for updating the tests to port.

[ghost]

Unfortunately the performance of the Peg.js parser is rather slow on lower-end CPUs due to the insane amounts of recursion here:

https://github.com/SNDST00M/vscode-textmate-languageservice/blob/v0.2.0/src/util/scope-selector-parser.pegjs#L7

I am hoping there is a way to get those running in WASM and I'll be looking into it this weekend.

EDIT: My laptop broke which takes me out of commission for 2 weeks minimum. I don't expect myself to solve this feature soon

[ghost]

Instead of WASM we can add caching logic to ScopeSelector class for match results, and this will work at 75% of the speed of the existing VS Code matchers! I will open a pull request for this soonish