Add language support for GYB

[fpseverino]

Add syntax highlighting and language configuration for Swift GYB (.swift.gyb) files.

Motivation:

Resolves #1455

Language support for GYB can't be implemented via sourcekit-lsp, as GYB can contain arbitrary Python.

The TextMate grammar can't be an injection grammar because GYB files have a different file extension from normal Swift files, and also because sourcekit-lsp would activate and throw a lot of errors.

Additions:

• [X] New swift-gyb language in package.json
• [X] Language configuration for Swift GYB with code folding, brackets definition and autoclosing
• [X] TextMate grammar for Swift GYB
• [ ] Icons for .swift.gyb files

Known issues:

• GYB code blocks (%{...}%) and control statements (% keyword: ... % end) are not always highlighted when they are inside Swift bodies (structs, extensions and funcs, in other words when not in the top level of the file) and do not have at least one level of indentation.
• Swift keywords are not always correctly highlighted when the preceding structure, function or variable name starts with a GYB block

[fpseverino]

Is there any way we could add tests for this at all?

Microsoft recommends using this package to test the grammar. I will take a closer look at it and try to add tests ASAP.

[matthewbastien]

Oh and I just realized that you'll need to add the new language files to the .vscodeignore or they won't make it into the VSIX. We ignore everything by default so that we're explicit in what we include:

!swift-gyb.language-configuration.json
!syntaxes/**

[fpseverino]

I added some tests using the package I mentioned previously. To run them in my dev container I use the command:

./node_modules/.bin/vscode-tmgrammar-test 'test/unit-tests/**/*.test.swift.gyb' -g test/unit-tests/syntaxes/swift.tmLanguage.json -g test/unit-tests/syntaxes/MagicPython.tmLanguage.json

What do I have to do to make the tests run automatically?

[matthewbastien]

What do I have to do to make the tests run automatically?

You'll want to create a new script in the package.json that runs the grammar tests and then add that to the test script so that it gets run in CI. Something like:

{
  "scripts": {
    // ...
    "test": "vscode-test && npm run grammar-test",
    "grammar-test": "vscode-tmgrammar-test 'test/unit-tests/**/*.test.swift.gyb' -g test/unit-tests/syntaxes/swift.tmLanguage.json -g test/unit-tests/syntaxes/MagicPython.tmLanguage.json"
    // ...
  }
}

[award999]

@fpseverino is the checklist above accurate as far as missing icons is all this PR is missing?

[fpseverino]

@fpseverino is the checklist above accurate as far as missing icons is all this PR is missing?

Yeah, there are still the known issues written in the description though, and more tests could be added I guess.

[fpseverino]

How can I resolve the merge conflicts and CI failures caused by package-lock.json?

[plemarquand]

@fpseverino While not ideal, the easiest way is to delete it locally and then run npm install then commit the changed package-lock.json.

[fpseverino]

I had to do a bit of juggling with Git but I think I managed it, may I ask when possible for another round of CI to make sure?

[award999]

How can I resolve the merge conflicts and CI failures caused by package-lock.json?

@fpseverino npm-merge-driver is a great tool to help with conflicts when pulling and rebasing

[award999]

@fpseverino is this still in draft state or can we mark it as ready for review?

[fpseverino]

@fpseverino is this still in draft state or can we mark it as ready for review?

I didn't mark it as ready for review because the known issues mentioned in the PR description are still unresolved. I think it's due to a mix of the limitations of TextMate grammars and my limited knowledge of them. You tell me if we can temporarily overlook these issues and start the review

[RedCMD]

including source.python does not apply the scopeName source.python https://github.com/fpseverino/vscode-swift/blob/9f8db0d2353b7bd1fc6c631a9e50c14b6af3e951/syntaxes/swift-gyb.tmLanguage.json#L68

you will need to target a scopeName that exists like the ones you have here: https://github.com/fpseverino/vscode-swift/blob/9f8db0d2353b7bd1fc6c631a9e50c14b6af3e951/package.json#L105-L107

the .+? will never match anything https://github.com/fpseverino/vscode-swift/blob/9f8db0d2353b7bd1fc6c631a9e50c14b6af3e951/syntaxes/swift-gyb.tmLanguage.json#L28

'test/unit-tests/**/*.test.swift.gyb'

does having no quotes test/unit-tests/**/*.test.swift.gyb work on linux? as it does work on windows

[fpseverino]

Hi @RedCMD, thanks a lot for the suggestions! Sorry for the delay

including source.python does not apply the scopeName source.python https://github.com/fpseverino/vscode-swift/blob/9f8db0d2353b7bd1fc6c631a9e50c14b6af3e951/syntaxes/swift-gyb.tmLanguage.json#L68

you will need to target a scopeName that exists like the ones you have here: https://github.com/fpseverino/vscode-swift/blob/9f8db0d2353b7bd1fc6c631a9e50c14b6af3e951/package.json#L105-L107

Unfortunately, however, I don't quite understand what I need to do to apply the scopeName source.python. Could you please help me with that?

does having no quotes test/unit-tests/**/*.test.swift.gyb work on linux? as it does work on windows

Seems to work on macOS, so I guess it will work on Linux too, we'll see in CI. Thanks!

[fpseverino]

The Windows check failure isn't my fault, right?

[award999]

@fpseverino no looks like a network timeout on setup. I've retriggered