Support more syntax tokens for theme configuration

[zarifpour]

Check for existing issues

• [X] Completed

Describe the feature

Proposed Solution:

1. Expand Theme Configuration: Extend the theme's configuration file to include more syntax tokens, enabling granular customization for different code elements.

2. Language-specific Support: Allow themes to define colors for syntax tokens specific to different programming languages.

This approach will enhance user experience by allowing more detailed and language-specific theme customization.

If applicable, add mockups / screenshots to help present your vision of the feature

No response

[maxdeviant]

@zarifpour Do you have specific examples of things that we don't currently support?

[zarifpour]

I can look for a better example, but here's a quick one. Let me know if it makes sense:

I don't think it's possible to color these specific elements, where in VSCode I can target these specifically and configure their colors and font style.

[drcongo]

I've been porting my personal Sublime Text theme to Zed and found a few things that I couldn't seem to set colours for...

Sublime

Zed

A quick list of Python tokens (though bear in mind that I may have just failed to find the right keys)...

• imported modules and functions get the same colour as classes / functions within the file
• Couldn't find a token to colour the dunders __all__ or __init__ (note that they're different used inside or outside a class too
• The “%(name)s” inside a string
• Function / method args
• self which is a big one for readability
• Exceptions like the TypeError in the screenshot
• Lots of things in the f-string

I think there's probably a few more too, these were just the ones visible in the screenshots of a random page from the Django codebase. Apologies if all these are actually possible and I'm just adding noise to the thread.

[pimmee]

Here's an example issue I created in the VSCode Dark Plus theme extension repo, which looks completely different in Zed vs VSCode.

When I tried to fix the colors and make a PR, I also quickly ran into issues with too few specific syntax tokens, especially for Typescript. Some examples include return or await. When I looked into how VSCode does it I recall seeing they have language specific scopes.

Would love if the color theming can be improved!

[codeitlikemiley]

I guess this is the same as my issue https://github.com/zed-industries/zed/issues/13961

I really need to make rust syntax highlighting more noticeable... its hard to work with all types having same color

[everdrone]

Also Rust attributes have inconsistent highlighting zed: vscode:

---

And Rust doc comments don't have any special highlighting. In vscode, rust doc comments get some flavor of markdown highlighting (links to crate items, types, functions, constants, with their respective color in the active theme), plus code blocks get rust syntax highlighting vscode:

[ehamiter]

This might also help cover my issue-- I'm unfamiliar with what order new LSPs/treesitters/syntax tokens occur to support colorization. Here's an example of a django template / html file that is in Sublime Text that is being highlighted:

...and here is an example of the same django template in Zed that is not being highlighted:

---

https://github.com/zed-industries/extensions/issues/1303

[demiurg]

Another obvious example seems to be Python decorators, which are not functions, declarations vs calling. Not sure how to differentiate those.

Scopes are very important for theming, here are some differences from neovim: https://gist.github.com/swarn/fb37d9eefe1bc616c2a7e476c0bc0316

I just started porting a theme to Zed, and.I am not able to even differentiate a from statement from a decorator.

[uncenter]

Expand Theme Configuration: Extend the theme's configuration file to include more syntax tokens, enabling granular customization for different code elements.

I'm not really sure what this means in the context of Tree-sitter queries, if you are referring to https://zed.dev/docs/extensions/languages#syntax-highlighting - that list is not complete and you can use any other common Tree-sitter capture you know of iirc.

Language-specific Support: Allow themes to define colors for syntax tokens specific to different programming languages.

This though is I think the core of this issue, or rather a separate issue from whatever the first one is. Scoping tokens to languages would be great. nvim-treesitter does this as follows:

As an additional rule, capture highlights can always be specialized by language, by appending the language name after an additional dot. For instance, to highlight comments differently per language:

hi @comment.c guifg=Blue
hi @comment.lua guifg=DarkBlue
hi link @comment.documentation.java String

(https://neovim.io/doc/user/treesitter.html#treesitter-highlight-groups)

[nicholaswmin]

At least one major for me is private properties in Javascript

class NumericalImpostor {
  #value = 0 // // <-- can't target private property
  constructor(initial) {
    this.#value = initial < -- same 
    this.nonprivate = 'this is democracy manifest!' //<-- all good, target with `property`
  }
  
  notifyCacheMiss() { 
    this.#value++
  }
  
  valueOf() {
    return this.#value
  }
}

const foo = new NumericalImpostor(10) + 10
console.log(foo)

Btw is there any actual rationale/body of work that says grouping certain keywords together under 1 color actually improves readability ?? Some tokens, It jsut feels like I'm aimlessly mashing crayons together.

[uncenter]

At least one major for me is private properties

What are you trying to do with it? It is parsed by the grammar as the private_property_identifier node, but isn't highlighted by the Zed highlights queries for JavaScript right now. You could add this query from Neovim to mark it as a @variable.member if you want:

https://github.com/nvim-treesitter/nvim-treesitter/blob/cfbbdd5effbde3d47d2d662e67cfcd62a93b783f/queries/ecma/highlights.scm#L13C1-L13C47

[nicholaswmin]

What are you trying to do with it? It is parsed by the grammar as the private_property_identifier

Highlight it I suppose...

You could add this query from Neovim to mark it as a @variable.member if you want:

that's so nice man thank you for digging it out for me on a Sunday

Can I add that query as an override in my own config or do I need to create an extension or mess with Zeds own files?

Also this leads to the point I'm asking. non-private members also fall under the same group @variable.member correct? If that's so what do I actually gain by grouping these 2 together?

might be too generic of a question but for some tokens, I understand the rationale of coloring them. ie code comments and perhaps strings benefit from highlighting, I can easily see myself tripping over them in non-coloured syntax - But what about the rest of the tokens? Is there any rule of thumb/guidelines etc? Or do we just try and paint every arbitrary category of a token a distinct arbitrary color and call it a day?

Edit:

nvm I found something to keep me occupied: https://gist.github.com/swarn/fb37d9eefe1bc616c2a7e476c0bc0316

[uncenter]

that's so nice man thank you for digging it out for me on a Sunday

No problem!

Can I add that query as an override in my own config or do I need to create an extension or mess with Zeds own files?

I don't think you can add it to your own config, no. You could PR it though if you want to, add it in to that file in this repository I linked above.

Also this leads to the point I'm asking. non-private members also fall under the same group @variable.member correct? If that's so what do I actually gain by grouping these 2 together?

Ah yeah looking at the nvim-treesitter file they also use @variable.member for the other properties - Zed uses it's equivalent @property for the same nodes, so you would want to use that here instead of variable.member - meaning you couldn't really differentiate it. However, if we look at Helix's supported captures - https://docs.helix-editor.com/themes.html#scopes - they have variable.other.member.private, specifically for this exact usecase! See https://github.com/helix-editor/helix/pull/10554 where it was introduced a few months ago. There is an ongoing issue of standardization of Tree-sitter captures though, so it is tricky figuring out what captures to use, and for variable.other.member.private specifically no existing Zed theme supports it (or even the less specific variable.other.member for that matter). So maybe this isn't a real possibility to add for now until more clarity on what captures we can use is given from the maintainers.

might be too generic of a question but for some tokens, I understand the rationale of coloring them. ie code comments and perhaps strings benefit from highlighting, I can easily see myself tripping over them in non-coloured syntax - But what about the rest of the tokens? Is there any rule of thumb/guidelines etc? Or do we just try and paint every arbitrary category of a token a distinct arbitrary color and call it a day?

The process goes as follows:

Text -> Parsed Syntax Tree ("Nodes") -> Capture Queries -> Theme Highlights

We are talking about the third step, capture queries, where you are correct - we just mark different syntax nodes (which are arbitrary and language/grammar specific) with generic/categorical "captures", and then the burden is on themes to specify which colors to use for which captures.

nvm I found something to keep me occupied: gist.github.com/swarn/fb37d9eefe1bc616c2a7e476c0bc0316

See https://github.com/zed-industries/zed/issues/5345 fwiw.

[nicholaswmin]

do these captures need to be baked into zed during compilation?

Why does Zed need to act as a middleman here? Wouldn't it be more flexible to have a set of basic tokens (as is now) and the rest can be overriden by the theme implementation if and when needed.

[uncenter]

do these captures need to be baked into zed during compilation?

Yes, but extensions can provide them too.

Why does Zed need to act as a middleman here? Wouldn't it be more flexible to have a set of basic tokens (as is now) and the rest can be overriden by the theme implementation if and when needed.

How would you imagine a theme overriding "if and when needed"? TextMate (used in VS Code) also has predefined "scopes" for each grammar that themes target, but it doesn't support themes arbitrarily marking up code (seems like that's what you are suggesting) as far as I know.

[nicholaswmin]

My experience here is very limited and Im not particularly smart either so rather than waste your time and go back/forth I think I'll go through the TreeSitter docs first - What I understood this far is that TreeSitter is generating some type of AST and that AST is targeted and categorized like you explained, thus exposing those categories for theming.

gist.github.com/swarn/fb37d9eefe1bc616c2a7e476c0bc0316

amazing yes, thats exactly the material I'm after. Semantic highlighting lists some reasons on how it helps the end-user and that's exacly what I want to readl.

Non-semantic highlighting is kinda baffling to me. What am I supposed to gain by coloring i.e: const and async the same? You might gain some structure if you're skimming through I suppose but that's it. And why is TextMate such a highly regarded "standard" that everyone points back to ? Are there actual visual merits to their specifications or did they create a superset of definitions that is simply common across many languages hence highly reusable?

The private property is a good example of this; sure both the private property and the regular property are both properties of some object, but it sounds more sensible to differentiate rather than group them on the basis of their visibility differences. Then again , for another person it might be the complete opposite and this could be purely a matter of taste in coloring.. i guess. Another example could be the == vs === operator, a common gotcha in less experienced JS people. In that case, colouring those 2 cases separately makes some sense to me at least; On the other hand, the current situation of coloring them identically makes no sense to me, at least for the time being..

..Not looking for specific answers here just the 1st time I'm doing own syntax highlighing and I 'm having a trouble finding a (even a little bit, not too much) of a reasoned approach to my choices.
I think i'll post this in the Discord channel if instead of going off on tangents and adding fluff to this isssue

[uncenter]

Non-semantic highlighting is kinda baffling to me.... why ?

Semantic highlighting is quite difficult. As mentioned in that gist, it requires an LSP just for it - writing a Tree-sitter grammar is much easier than writing an LSP, for one. And Tree-sitter is quite accurate and I find it very helpful :)

what did I just gain by coloring i.e: const and async the same? You might gain some structure if you're skimming through I suppose but that's it?!?

Are we talking about the literal keywords with the text const and async, or variables and functions defined with those keywords? The later is what semantic highlighting changes, not the former. You gain basic information about what is a variable, what is a function, etc. Constants are also colored differently in most cases, fwiw. I do agree that semantic highlighting can be useful, but I strongly disagree that you gain nothing by using just non-semantic highlighting with Tree-sitter.

And why is TextMate such a highly regarded "standard" that everyone points back to ? Is itn just because they defined a structured format efirst? Are there actual visual merits to their specifications or did they create a superset of definitions that is simply common across many languages hence highly reusable? Srhould I color a particular keyword more prominently than others for some reason?

I don't regard it highly, I think it is an awful way to work with syntax highlighting. Using lists of regex patterns to match on was actually harder to write in my experience than a full, complete Tree-sitter grammar/parser. I do refer to it though because VS Code is the most popular editor in the world, so for sake of comparison it is a good example. (Side note: keeping an eye on https://togithub.com/microsoft/vscode/issues/50140!)

The private property is a good example of this; sure both the private property and the regular property are both properties of some object, but it sounds more sensible to differentiate rather than group them on the basis of their visibility differences. Then again , for another person it might be the complete opposite and this could be purely a matter of taste in coloring.. i guess.

I think it completely makes sense to differentiate them. And as we know, we can! The issue I mentioned previously is that the captures we should use to mark that are unclear. The Tree-sitter ecosystem is still in its early stages, and the editor compatibility/standards aren't quite there yet. At some point, I hope there is an established capture for private variable members that we can use here!

Another example could be the == vs === operator, a common gotcha in less experienced JS people. In that case, colouring those 2 cases separately makes some sense to me at least; On the other hand, the current situation of coloring them identically makes no sense to me, at least for the time being..

To be fair I've never heard of these being differentiated even via semantic highlighting, so this might just be a you thing ahah. Personally I use a font with ligatures that differentiates the sizes of the bars in == and ===, so that the triple equals is clearly larger. You can also just use a linter that warns on == :)

..Not looking for specific answers here just the 1st time I'm doing own syntax highlighing and I 'm having a trouble finding a (even a little bit, not too much) of a reasoned approach to my choices. I think i'll post this in the Discord channel if instead of going off on tangents and adding fluff to this isssue

I can't quite tell but, by "doing own syntax highlighing" do you mean you are writing a theme? Happy to help out and I agree we might want to leave this thread alone - feel free to dm me on Discord @uncenter.

[h-a-n-n-e-s]

One more differentiation that would be nice (and afaik has not been mentioned), distiguishing function parameters (like "voice" in the screenshot below) and normal variables.

[uncenter]

One more differentiation that would be nice (and afaik has not been mentioned), distiguishing function parameters (like "voice" in the screenshot below) and normal variables.

I believe that is called semantic highlighting; see https://github.com/zed-industries/zed/issues/7450.

[jasonkuhrt]

In TypeScript @type captures const. This seems like a bug. I expected const ... to be @variable.

[jasonkuhrt]

I would like to be able to target exported nodes. So for example in export const a = 1 I would like control over:

1. keyword.export
2. keyword.variable.exported
3. variable.exported

Going further, I would like to see the ability to target in this case too:

const a = 1

export { a }

I still want const a = 1 to be matched as above.

I suspect there are lots of upstream reasons to say this is not easy, but one can hope and share.

The notion of syntactic themes empowered by hyper-granular selections in turn empowered by semantic analysis tooling is super interesting to me. I think Zed + TS Go make a compelling combination to posit a world in which this could become reality too.

[hisbvdis]

Please add syntax highlighting tokens for

• function declaration
• JSX Components
• control keywords (import/export, if/else)