Add semantic syntax highlighting support

[aripiprazole]

Closes #7450

Hello, good evening, this is my first PR for this project, and I'm still working on it, I just wanted to show what I did until this moment:

• [x] Proof of Concept implemented
• [x] Settings that enable and disable semantic tokens (inspired by inlay hints)
  • [x] Language-specific setting
• [x] Request to the server only when the request SemanticTokensRefresh is received
  • [x] Fix crashing and make it faster
• [x] Refresh semantic tokens when you open a file with a language server
• [x] Refresh semantic tokens when you switch the language of the file/or when you emit the Reparse event
• [x] Request all capable language servers so we can implement some extensions like TODO Tree in VSCode for Zed with just language servers
• [x] Enable to toggle just to one file at once, because some files are too big and it's a good feature.
  • [x] Button to toggle in UI
  • [x] Event to toggle the button to bind with a keyboard setting
• [x] Fix crashing when unsupported tokens come
• [x] Proto implementation in the LSP command
• [x] Create another iterator to semantic highlighting instead of using CustomHighlightings
• [x] Fix not showing highlights with 1-len chars highlighted
• [x] Customize modifiers to specified tokens in the settings
• [x] After implementing cache, the focus action is being triggered, and all the token colors are being erased, need a fix
  • [x] Implement ~delta requests/~range requests It's useful in large files like 18k lines that we don't require all the tokens, only the visible tokens, so it's a must-be for this PR (it's lagging a lot, really)
  • [x] Create a response cache from the LSP, and add tokens incrementally there when writing, so we can avoid flicking on the screen
• [x] Fix the range tokens that are trying to be relative to the start of the file, instead of relative to the requested anchor

Test state:

• [ ] Cache tests
• [x] Renderization tests
  • maybe render the token text, and test better the stuff in the token_map.rs
• [x] Requests/parsing delta from server tests

Implementation details:

• I'm firmly basing the implementation on the inlay hints implementation
• ~~I didn't create a cache for semantic tokens, because it doesn't add anything new in the UI, even with range/delta requests, I think it should not need a cache because a simple vec would do all the work~~ (I got the reason why inlay hints have cache when I opened a big file with semantic tokens enabled)

Other Goals/Questions

• ~~Create another chunk iterator to the semantic highlighting~~ (If I split it will be better for testing, so I will)
• ~~Support range requests (I think this one should be another issue/PR). It's added the SemanticTokensRefreshReason so we can implement this better in the future~~ (It will be blinking the colors if I don't implement)

Consideration regarding existing themes:

• Maybe add a modding interface on top of them to create pleasing semantic tokens in the default
• Should I care about it or expect developers to change their themes?
• Merge semantic_tokens.rs into lsp_command.rs?

Release Notes:

• Added two new sections in the theme configuration
• By default, the semantic tokens are made by associating the existing syntax tokens of tree-sitter
• Use the chunk iterator and the CustomHighlightsChunks at the display map
• Added Full Request to LSP (so it requests all the tokens to a file when refreshed)

I'm currently testing with the theme https://github.com/PyaeSoneAungRgn/github-zed-theme/pull/23

Before

After

I'm unable make the theme beautiful sorry

[youngxhui]

Thank you for your work. I have been looking forward to the feature of highlighting semantic tokens for a long time.

[saiintbrisson]

amazing work 🙏

[Angelk90]

@aripiprazole : Is this effect here the theme that implements it?

[aripiprazole]

@aripiprazole : Is this effect here the theme that implements it?

its the indent rainbow on the zed settings:

{
  "indent_guides": {
    "coloring": "indent_aware",
    "background_coloring": "indent_aware",
    "active_line_width": 2
  },
  ...
}

[Angelk90]

@aripiprazole : Thank you, I thought they hadn't implemented it yet.

https://github.com/zed-industries/zed/issues/5259#issuecomment-2284059594

[DotRed108]

You are a legend.

[aripiprazole]

Just need to finish the tests, so I'm opening to review already

[clicktodev]

can we get rainbow brackets while we're at it? thanks!

[aripiprazole]

can we get rainbow brackets while we're at it? thanks!

sure, I'll see whats happening

[clicktodev]

can we get rainbow brackets while we're at it? thanks!

sure, I'll see whats happening

thanks! linking this for relevance https://code.visualstudio.com/blogs/2021/09/29/bracket-pair-colorization

[aripiprazole]

can we get rainbow brackets while we're at it? thanks!

sure, I'll see whats happening

thanks! linking this for relevance code.visualstudio.com/blogs/2021/09/29/bracket-pair-colorization

I was a little dumb, I thought it was an error because I just thought that rust analyzer had bracket pair colorization, but you don't think it should be the servers duty to do the colorization, or another feature?

[aripiprazole]

i added support for multiple tokens superposing, it should be easy to create another LSP to highlight the brackets, too

[clicktodev]

i think i misunderstood the purpose of this pr. i made the comment about rainbow brackets because i thought this pr was about adding colored indentation

[aripiprazole]

I'm still trying to find out why the server isn't being requested in the tests.

[aripiprazole]

this branch of the code is not being reached, and I'm not sure why (the code inside the cx.spawn)

[pedrocarlo]

Thanks for the work! This is one of the things that is blocking me from using zed.

[aripiprazole]

Thanks for the work! This is one of the things that is blocking me from using zed.

theres a couple of work i need to finish, basically stop blinking again, because i discovered some edge cases, and i'm rethinking part of the logic/working these last days, so i'm taking a bit longer to finish, but as soon enough we're going to have it :)

i was willing to use zed with lean, but lean only has semantic highlight

[Anthony-Eid]

Hey, I would love to use this to improve the accuracy of debugger inline values. Do you mind if I jump in sometime next week to help push this PR across the line?

I should have a couple of hours to spend on this at least, so please let me know.

[aripiprazole]

I'll return try to go back to work on this this weekend, feel free to contribute if you want, just lmk if you want to try to work in pair. I got sick and started a new job

[kontheocharis]

Hi there, thanks for all your work. I would really love to have this feature. What else remains to be done? I can dedicate some time to this if you are too busy :)

I see two unchecked todos:

• Fix the range tokens that are trying to be relative to the start of the file, instead of relative to the requested anchor
• Cache tests

Are there any other outstanding tasks or is that mostly it?

[aripiprazole]

Hi there, thanks for all your work. I would really love to have this feature. What else remains to be done? I can dedicate some time to this if you are too busy :)

I see two unchecked todos:

• Fix the range tokens that are trying to be relative to the start of the file, instead of relative to the requested anchor
• Cache tests

Are there any other outstanding tasks or is that mostly it?

Theres the pipeline of rendering, which is taking the most of the time, and the first one is almost done, can you work on the cache tests and send me a PR? Thanks for helping as well, I really wanna use this feature as well

[aripiprazole]

I'm more advanced on finding out what's wrong with the rendering pipeline which is the function TokenMap::sync and TokenMap::splice, and the functions related, but I have a bunch of experiments, and a lot of uncommited files here, if you wanna do this part as well, it would be good

If you need, i can create issues as well referencing the files @kontheocharis

[aripiprazole]

Gotta update the PR description with a simpler one and let this progress stuff into the local repository as well, so I can create issues and accept PRs freely. I think it was a bad idea trying to do one big PR for this feature

[Anthony-Eid]

Just an update from my end; this PR is definitely something I want to jump in on and help, but the debugger work has been taking up most of my time.

I'm going to dedicate some time to this on Friday and this weekend to get familiar with the changes, and I'll hopefully be able to help out a bit next week. Sadly, I can't guarantee anything because shipping the debugger is my main priority at the moment.

Also, @aripiprazole, congrats on the new job 😁

[kontheocharis]

I'm more advanced on finding out what's wrong with the rendering pipeline which is the function TokenMap::sync and TokenMap::splice, and the functions related, but I have a bunch of experiments, and a lot of uncommited files here, if you wanna do this part as well, it would be good

If you need, i can create issues as well referencing the files @kontheocharis

I will start with the tests; is it okay to open an PR on your fork when I'm done?

[aripiprazole]

I'm more advanced on finding out what's wrong with the rendering pipeline which is the function TokenMap::sync and TokenMap::splice, and the functions related, but I have a bunch of experiments, and a lot of uncommited files here, if you wanna do this part as well, it would be good If you need, i can create issues as well referencing the files @kontheocharis

I will start with the tests; is it okay to open an PR on your fork when I'm done?

of course, thanks

[SomeoneToIgnore]

To chime in, I'm not very happy with the inlay_hint_cache.rs existing in each editor and plan to move the inlay hint LSP data into lsp_store.rs, make it more centralized and shared between editors.

The logical next step would be to extend it with the LSP semantics highlighting data, and other LSP data later so that there is a single tree-like structure to hold it all, with the same "get or fetch" pattern implemented once for each kind of data.

It's great to see the prototype with the inlay_cache copied instead, but let's keep in mind that it's not the final state of the related code — and avoid extra work there if possible.

[aripiprazole]

hey @Anthony-Eid i think its quite okay now the PR and ready to review!

[max-frai]

This will be huge, even better than debugger

[aripiprazole]

forgot to remove the simd patch from the cargo.toml that i was using bc smh it wasn't building here