Deprecate the 'semantic_tokens' config option in favor of 'augmentsSyntaxTokens'

[Techatrix]

The augmentsSyntaxTokens client capability can be used to indicate that the client is going to combine existing syntax tokens with the semantic tokens of the server. This can not only reduce the amount of data that is transferred over the protocol but can also lead to better highlighting as syntax tokens often more accurately distinguish between different language constructs on the syntax level. An example would be keywords where the LSP protocol has a single standard token type compared to syntax highlighting where different keywords are often differentiated between based on their purpose.

The semantic_tokens config had been added for this purpose. The partial option is used to prevent LSP semantic tokens to override existing syntax highlighting of the editor. The benefit of the client capability is that this decision is automatically made by supported editors instead of relying on user to select the right configuration.

The following editors should be unaffected:

• VS Code: enables partial by default
• LSP4IJ (JetBrains): does not enable augmentsSyntaxTokens
• Helix: does not use semantic tokens
• Zed: does not use semantic tokens
• Kate: does not enable augmentsSyntaxTokens
• Eglot (Emacs): TODO
• lsp-mode (Emacs): TODO

The following editors are affected:

Sublime Text

Requires the semantic_highlighting config option to be enabled.

Before: After:

The new output more closely matches the result when disabling semantic highlighting:

Neovim

Tested with tokyonight.nvim

Before:

After:

The zig.vim plugin maps the var, const, comptime and threadlocal keywords to Function which doesn't seem right.

The changes in Neovim should be investigated further before merging this PR.

[angelozerr]

@Techatrix I suggest thzt yiu create an issue on Lsp4ij to take care of this capability and if you could explain me how to implement it. Thanks!