implement semantic highlighting

[WebFreak001]

See https://github.com/microsoft/vscode/wiki/Semantic-Highlighting-Overview

would need proprietary serve-d RPC extension for now, but would be useful to consistently highlight functions as functions, etc.

[HuskyNator]

Given the existence of this issue, is it helpful to report current highlighting issues?

For example: I noticed an if statement with embedded '()' entries shows different color when linebreaks are introduced:

if((someFunc())
    || (someFunc())){
    writeln("test");
}

The second someFunc here takes variable.parameter.d instead of entity.name.function.d

I'm asking as I'm guessing things like this wouldn't really be an issue when semantic highlighting is used.

[MonoMaxMW]

@WebFreak001 what would be required to get this going? I am willing to help with this topics.

[WebFreak001]

@MonoMaxMW I have written down pretty much everything that came to mind that would help you (or anyone else) implementing this feature, as well as other related features. Don't be afraid by the large list, if you read from top to bottom and don't skip over the items too quickly, it should be a straightforward process to apply to the development process I think. If you have any further questions, feel free to ping me here, on discord or on matrix.

1. in serve-d in extension.d you need to tell in the registration options that serve-d offers semantic highlighting
   • this is called semanticTokensProvider and is part of the server capabilities
   • you can just add a new entry above and within https://github.com/Pure-D/serve-d/blob/ce3d7f4ac3c638679c90c426d970a117c2d361a7/source/served/extension.d#L287-L307
2. inside source/served/commands/, reuse an existing registered module OR create a module that you will need to register in extension.d (it's a list of modules)
   • if you make a new module you need to import it here: https://github.com/Pure-D/serve-d/blob/ce3d7f4ac3c638679c90c426d970a117c2d361a7/source/served/extension.d#L33-L52
   • AND register it here: https://github.com/Pure-D/serve-d/blob/ce3d7f4ac3c638679c90c426d970a117c2d361a7/source/served/extension.d#L1122-L1146
   • in a new module add an import for import served.types; (it contains almost everything needed)
3. the protocol defines that the client pulls the tokens from the server, so you just need the define a method that handles the requests from the client (the editor) and responds with a list of tokens for the requested range
   • for every method you define, you create a method such as

     @protocolMethod("textDocument/semanticTokens/full")
     SemanticTokens provideSemanticTokensFull(SemanticTokensParams params)
     

   • the method name is irrelevant, the UDA protocolMethod, along with the module being registered in extension.d, makes serve-d automatically call this method and handle the JSON (de)serialization for you
   • you may need to adjust the response according to what the editor can actually support - this is part of TextDocumentClientCapabilities, the member is called semanticTokens and you can find the docs about its members here (the struct should be defined in serve-d already, inside protocol.d you can see the definition)
   • to read the capabilities, there is a global variable defined in serve-d types.d that gets initialized in the initialize method in extension.d
   • semantic tokens are documented and explained here
   • check the docs for what protocol methods you can and/or must provide
4. for the actual tokens there are a few data sources to consider:
   • libdparse's getTokensForParser will give you raw tokens, which as a start would make it possible to highlight things such as strings and keywords (stopping here would be enough for a first implementation)
   • eventually this will get slow for big files and the editor will appear to lag - this is my primary concern with the implementation right now and will need a persistent tokens cache for each document in the future
5. Future thoughts:
   • in a future PR using these tokens you can then call parseModule to actually have it be parsed as D code and get an AST that we can visit using a visitor to mark a bunch of important things, such as variable declaration types, parameter types and return types
   • this too will be a point that needs major optimizations, both for RAM and for CPU usage, which I'm also currently working on in the libdparse repository
   • in an even later future PR we can then also use either serve-d/workspace-d's symbol index or dsymbol (through DCD) to look up e.g. template parameters or just every type, to check if they are aliased or to check if they are types or manifest constants or any other variables or any other symbol kinds
   • note for what I was planning here
     • eventually I'll port the additional symbol index capabilities from serve-d to dsymbol (e.g. some lookup algorithms as well as crude find references help structures that I'll still need to refine)
     • the module parsing is something that may be cached as well
     • the dsymbol index might eventually just be part of serve-d, like others have suggested with embedding DCD
     • a better event-based scheduler for serve-d is high on my to-do list that will help making this feature actually feasible for large code bases, as well as all other features and in general make serve-d respond better, with higher performance, less lag and less CPU usage