Support advanced scope selectors

[keith-hall]

• [x] multiple exclusions
• [x] empty scope selectors always match
• [x] pipes are aliases for commas
• [ ] parenthesized sub-matches
• [ ] ampersand operator (and operator precedence)

---

Currently, syntect doesn't support grouping parens or & operators in scope selectors, which means it doesn't color scopes correctly when the .tmTheme file being used contains these more advanced constructs.

See the update under "One last twist" at http://blog.macromates.com/2005/introduction-to-scopes/ for a brief example

it's also possible to AND, OR, and subtract scope selectors, e.g.: (a | b) & c - d would select the scope which is not matched by d, and matched by both c, and a or b.

I've never really found any good documentation on scope selectors, so I tried to write a bit more about them on SO, which I'll link to here in case it's useful: http://stackoverflow.com/a/37407588/4473405

EDIT: I also found http://textmate.1073791.n5.nabble.com/formal-definition-of-scope-selector-syntax-td12109.html just now, which seems to indicate that | and , don't have equal precedence.

As a simple example, if we imagine the following Python code:

def test:
    global abc

then the selector (storage | entity.name) & meta.function should match def and test.

Please would you consider adding support for this to your awesome syntect library.

[trishume]

I agree it's a useful and neat feature, but I must admit I don't predict I'll find time or motivation to do it myself. Other functionality like #34 seems more important in that it enables new use cases and prevents basic usage from failing horribly (see #33). This functionality just allows marginally nicer themes.

If someone else (like you) wants to take up the task though I'm happy to provide advice, code review, guidance and information.

[FichteFoll]

For the record, it appears that #39 added support for pipes (same as commas) and multiple exclusions. What remains are parentheses for sub-matches and ampersands for AND-connections, as far as I can see. Maybe add a checklist to the OP?

[keith-hall]

Correct - I've now updated the OP with the checklist, as suggested.

[keith-hall]

Note that the Monokai color scheme that Sublime Text (build 3156) ships with includes the following scope selector:

{
    "name": "markup punctuation",
    "scope": "text & (punctuation.definition.italic | punctuation.definition.bold | punctuation.definition.raw | punctuation.definition.link | punctuation.definition.metadata | punctuation.definition.image | punctuation.separator.table-cell | punctuation.section.table-header | punctuation.definition.constant)",
    "foreground": "color(var(white) alpha(0.67))"
},

so if people were to try to use that color scheme in syntect (currently impossible, because syntect doesn't support the sublime-color-scheme format), then their code/text could look different to how it does in Sublime.

[brupelo]

Any plans for this one? Just tested the examples provided here with syntect and it seems not all operators are supported by syntect:

keyword.control.php                               keyword                                           MatchPower(1.0)
keyword.control.php                               keyword.control                                   MatchPower(2.0)
keyword.control.php                               control                                           None
keyword.control.php                               keyword.cont                                      None
keyword.control.php                               keyword.control.php.embedded                      None
source.php meta.block.php keyword.control.php     keyword                                           MatchPower(64.0)
source.php meta.block.php keyword.control.php     meta                                              MatchPower(8.0)
source.php meta.block.php keyword.control.php     keyword meta                                      None
source.php meta.block.php                         text | meta                                       None
source.php                                        text, meta                                        None
source.php meta.block.php keyword.control.php     keyword & meta                                    None
source.php meta.block.php                         keyword & meta                                    None
source.php meta.block.php                         source - keyword                                  MatchPower(1.0)
source.php meta.block.php keyword.control.php     source - keyword                                  None
source.php meta.block.php                         source - (keyword | storage)                      MatchPower(1.0)
source.php meta.block.php                         (source - source.php) | text                      None

Btw, If I'm not mistaken it seems both scopes & selectors are just simple rules coming from a simple grammar such as:

atom:         «string» | '*'
scope:        «atom» ('.' «atom»)*
path:         '^'? «scope» ('>'? «scope»)* '$'?
group:        '(' «selector» ')'
filter:       ("L:"|"R:"|"B:") («group» | «path»)
expression:   '-'? («filter» | «group» | «path»)
composite:    «expression» ([|&-] «expression»)*
selector:     «composite» (',' «composite»)*

Summing up, right now the grammar covered by syntect seems to be a very simple subset from the most general one.

[keith-hall]

out of curiosity, where did you conjure up that grammar from? it contains some tokens which don't exist in Sublime Text or TextMate...

[brupelo]

Check out prymatex project from diegomvh on github, really amazing textmate python clone project full of features, although it needs some serious refactoring in many places and its difficult to run on windows.

Related... I've tested prymatex parser in comparison to syntect's and the former is much slower. Summing up, syntect is quite fast but incomplete and prymatex is complete but slow... Also, related to this subject, there is also edbee-lib alternative, really good app but also needs some serious refactoring.

I assumed that grammar would be the original textmate one... Is not?

[brupelo]

Related to this thread... I've created this little test where you can see how syntect doesn't support these basic operators.

Examples took from SublimeText docs