microsoft
[Bug] cmake syntax highlighting is broken
Brief Issue Summary
After update into 1.21.36(latest) version, syntax highlighting is not working well.
Before update (1.20.53)
After update (1.21.36)
CMake Tools Diagnostics
n/a
Debug Log
n/a
Additional Information
FYI, The color theme was Dark Modern. I used ssh with Windows host PC, working on Linux server.
Hi @shkwon98, thank you for reporting this issue. We think this issue is by design. We have refactored and improved the coloring language service support. Please refer to these issues https://github.com/microsoft/vscode-cmake-tools/issues/4367 and https://github.com/microsoft/vscode-cmake-tools/pull/4433. If we missed something, please let us know. Thank you.
Thanks for the reply @yanghhhhhhh . Well, If it was an intentional update, I have nothing to say about it, but this change is disappointing in some points.
line 7: project command
Why are different colors applied to the VERSION and LANGUAGES keywords? They are both options for the project command.
line 35: if command
Why are different colors applied to the AND and OR keywords? They are both logic operators for the if command. If the reason is because of the parentheses, it is even more awkward that the same grammar keyword has a different color just because of the parentheses. Furthermore, I think it would be right to apply the same colors to Comparisons/Version Comparisons as well.
line 40-49: option command
Is there a reason why the same color is applied to variable and value without distinguishing between them?
@shkwon98 Hi , thanks for the detailed information.
Reproduced this issue on Visual Studio v1.101.2 Code + CMake Tools v1.21.36 release version. Repro Steps: 1. Clone the project from https://github.com/bulletphysics/bullet3. 2. Open CMakeLists.txt and observe the code syntax highlighting.
FYI @gcampbell-msft
I've also noticed some incoherent syntactic colorations. Before the update, there were only a few keywords that weren't recognized, but now it seems to be worse. The theme is "On Dark Pro".
BEFORE update :
AFTER update :
In particular, some keywords (e.g. CHECK_START, TOLOWER) are the same color as the strings.
In some cases, I even have 3 different colors for the variables (green, blank, blue) :
Hi @josephgarnier, thanks for the information. Our team will investigate this issue sooner, and updated the progress here. Thanks for help us build a better CMake Tools!
@shkwon98 @josephgarnier @yanghhhhhhh This is expected in the sense that we recently made some updates to the language service. However, it's certainly not intended to have worse support. From looking at your before v after screenshots, I would agree that there are some minor things that have slightly better recognition, but it does seem like it's a slight degradation.
@shkwon98 @josephgarnier If possible, it'd be great to have you share your repros for us to have as tests to analyze against as we work to improve this. Let us know if this is a blocker to your workflow or overly distracting
@gcampbell-msft Sure, my screenshots comes from these 2 CMakeLists.txt:
- https://github.com/josephgarnier/cmake-modules-collection/blob/main/CMakeLists.txt
- https://github.com/josephgarnier/cmake-modules-collection/blob/main/tests/data/CMakeLists.txt
I noticed that the update adds new TextMate rules, could the cause be the theme that should be updated?
@gcampbell-msft Sure, my screenshots comes from these 2 CMakeLists.txt:
- https://github.com/josephgarnier/cmake-modules-collection/blob/main/CMakeLists.txt
- https://github.com/josephgarnier/cmake-modules-collection/blob/main/tests/data/CMakeLists.txt
I noticed that the update adds new TextMate rules, could the cause be the theme that should be updated?
It's definitely possible that the theme could be updated to make it appear more consistent. If you take a look at my change, I was attempting to make the "name" fields of the definitions to more closely match with what they actually are, and ones that would have default coloring schemes.
This may also be what is introducing the inconsistent appearance, as I may have changed the names such that they don't have the same colorization (or now they do).
However, looking at your repros, I do believe there are some inconsistencies present that are strictly due to the grammar (unfortunately).
@josephgarnier @shkwon98 I'd want to do more testing, but I do think I've possibly been able to refactor and address some of your concerns. I'll put some screenshots below addressing each one compared to what I've been able to create locally.
@shkwon98
"Why are different colors applied to the VERSION and LANGUAGES keywords? They are both options for the project command."
This is likely due to the difference in semantic versus syntactic highlighting. We don't current implement semantic colorization, and from what I can find in the CMake documentation I don't think LANGUAGES is a syntactic keyword, but a semantic one.
line 35: if command
I was able to locally modify and get this:
line 40-49
Still needs investigation
@josephgarnier It's a little harder to get specific screenshots, but I think I have your scenario somewhat improved as well.
@josephgarnier @shkwon98 While we investigate this further, if you have time or interest to test it and report back, that could be very helpful:
You'll need to download, modify the file extension to .vsix, and then install manually. Thanks!
@gcampbell-msft With the exception of the colors, that depend of the theme, I have almost the same syntax analysis as you.
There are still some strange things, as in lines 52 and 53: the command message has the same color as the STATUS keyword, but TOLOWER has the same color as a string value.
Here is the token of STATUS:
And here is the token of TOLOWER:
(For your information, because I don't know if this is a mistake, the version you gave me is 1.13, which is an old version)
@shkwon98 @josephgarnier @yanghhhhhhh This is expected in the sense that we recently made some updates to the language service. However, it's certainly not intended to have worse support. From looking at your before v after screenshots, I would agree that there are some minor things that have slightly better recognition, but it does seem like it's a slight degradation.
@shkwon98 @josephgarnier If possible, it'd be great to have you share your repros for us to have as tests to analyze against as we work to improve this. Let us know if this is a blocker to your workflow or overly distracting
@gcampbell-msft Sorry for the late reply. Mine was from here: https://github.com/nlohmann/json/blob/v3.11.2/CMakeLists.txt
@josephgarnier @shkwon98 While we investigate this further, if you have time or interest to test it and report back, that could be very helpful:
You'll need to download, modify the file extension to .vsix, and then install manually. Thanks!
@gcampbell-msft I tried using the extension you provided and it is better than before, but there are still some things I am disappointed about.
The colors applied to each cmake command and each command's options are not consistent, making it difficult to know which rule is being used.
This is likely due to the difference in semantic versus syntactic highlighting. We don't current implement semantic colorization, and from what I can find in the CMake documentation I don't think LANGUAGES is a syntactic keyword, but a semantic one.
Take a look at an example of a cmake project below:
There are also keys that have multiple values, like the LANGUAGES option. Since the colors of the options and values are the same, they cannot be completely distinguished by spacing or line breaks. This may not be a problem for experienced users who are familiar with cmake. However, I am concerned that if we go in this direction, the barrier to entry for cmake will be high.
Furthermore, I fundamentally disagree with the idea that the LANGUAGES option is only semantic. This option does the same thing as using the enable_language command. I actually use this feature to integrate Go language support into cmake by writing files like CMakeDetermineGoCompiler.cmake, CMakeGoCompiler.cmake.in, etc. So, what I want to say is that the LANGUAGES option is not just a way to help others understand the project configuration, but it is also an element that plays a role in the structure and grammar of a cmake project for the compilation process.
Thank you for the thoughtful feedback and detailed examples. Based on the regressions highlighted and the impact on usability, we've decided to revert the recent changes to CMake syntax highlighting for now. Please see this PR: Revert "Refactor and improve colorization language services support" by gcampbell-msft · Pull Request #4521 · microsoft/vscode-cmake-tools
That said, this area remains important to us, and we welcome contributions to help improve it further. As an open-source extension, we encourage community collaboration — especially on nuanced areas like language grammar and highlighting behavior. If you're interested in helping refine the grammar rules or improve semantic consistency, we would absolutely support and review contributions in this space.
I'm the maintainer of the vim-cmake-syntax-highlight-plugin (https://github.com/pboettch/vim-cmake-syntax/).
Over the years I thought how to extract information about keywords/commands and so from CMake's source-tree, not wanting it to do it manually. The way earlier maintainers and I found was to parse the output the different help-commands of cmake itself. This is far from perfect, but close. It helps to get a list of commands, modules, keywords per command. Which is sufficient to get command-context based syntax-highlighting.
I would be able to spend some time on this extension to see whether I and this method can help.
But I'm totally new to vscode-extensions. I'm don't even know where to start.
Maybe it would be faster to start a new empty syntax-highlighting project for testing and then see whether it can be merged here?
What do you think?
@pboettch While awaiting an official response, to test your changes locally, you can edit the file .vscode/extensions/ms-vscode.cmake-tools-1.21.36/syntaxes/CMake.tmLanguage.json.
I suppose your method is similar to the one used by the CMake extension. You have an example of an update pipeline on this PR.
Automatic extraction from CMake is import as with each release there are keywords and functions added and removed.
My method is similar, but scripted. And I have command-properties on a command basis. E.g. STREQUAL would only highlight if in an if()-command.
I think a viable solution (until Kitware creates maybe a real grammar) is to extract from the help (or however) and generate the textMate-grammer with a nice template.
@pboettch We also do a similar type of thing to extract information about commands and other things that CMake provides, because as you said, it makes it a lot easier to get updates without having to do it manually.
However, it sounds like maybe there is room for improvement in how that information gets translated to the language grammar? Is that what you're suggesting?
Yes, I'm new to textMate, but not the vim's highlight-grammar (if that can be called like this). What can be easily added here is a property per command highlighting. That is, the argument body of a command would only highlight acceptable keywords. The same for built-in variables vs. user-defined variables.
My example is STREQUAL would only highlight in a if-statement (if is a command in cmake) but not in add_custom_command.
I will look into textMate-grammer and vscode-extensions - and share once I'm happy.
I started to have a look, whether there is good coloring depends a lot on the theme. Most of the themes do not highlight all scopes differently: Here's a screenshot of my little progress, done half-manually/half-scripted using Monokai-dimmed.
It is much easier than vimscript, that's what I can already tell ;-)
Reading your PR, I wonder, do you work on he creation of new syntax analysis rules or only on coloring ?
As for the analysis, the official version has inconsistent results, and I wonder if you were able to correct them.
- Example 1:
list(POP_BACK CMAKE_MESSAGE_INDENT)
list(APPEND CMAKE_MESSAGE_INDENT " ")
The textmate scope of POP_BACK is string.unquoted.cmake and source.cmake, while the textmate scope of APPEND is support.function.commandProperty.cmake and source.cmake.
- Example 2:
set(expected_result
"apple"
"banana"
"orange"
"pineapple"
"carrot"
"strawberry"
"pineapple"
"grape"
"lemon:watermelon"
)
set(map "fruit 1:apple" "fruit 2:banana" "fruit 3:orange" "fruit 4:orange" "fruit 5:green:lemon" "INVALID" ":INVALID")
The textmate scope of expected_result is entity.name.function.cmake and source.cmake, while the textmate scope of map is string.unquoted.cmake and source.cmake (no scope for variable ?).
- Example 3:
if((NOT EXISTS "${folder}") OR (NOT IS_DIRECTORY "${folder}"))
The textmate scope of OR is entity.name.function.cmake and source.cmake.
- Example 4:
FetchContent_Declare(
cmake_test
GIT_REPOSITORY https://github.com/josephgarnier/CMakeTest.git
GIT_SHALLOW on
GIT_PROGRESS on
EXCLUDE_FROM_ALL
SYSTEM
STAMP_DIR "${${PROJECT_NAME}_BUILD_DIR}"
DOWNLOAD_NO_PROGRESS off
LOG_DOWNLOAD on
LOG_UPDATE on
LOG_PATCH on
LOG_CONFIGURE on
LOG_BUILD on
LOG_INSTALL on
LOG_TEST on
LOG_MERGED_STDOUTERR on
LOG_OUTPUT_ON_FAILURE on
USES_TERMINAL_DOWNLOAD on
)
The textmate scope of "on" is entity.name.function.cmake, not constant.language.boolean.cmake.
Note: I also noticed a coloring conflict when the CMakeTool and CMake extensions are enabled simultaneously. This is understandable, but it's something to keep in mind when reporting bugs (which I didn't do before...).
I think my grammar is more fine grained as to which keyword is allowed per function, e.g..
I was not looking into fixing the existing grammar as it was too limited for what I wanted to have. However, it was still really helpful for me to understand how textMate is working.
With the version of the grammar I created (#4527) your examples are showing as follows:
I don't have support for FetchContent - modules are on my todo list.
There is one thing I learnt during the development: the theme can take control over the syntax-scope. For example for punctuation in my case.
Where did you see the support of maps in CMake? I'm not aware of their existence.
Here are my scope inspections.
Wow, that's great work!
Unfortunately, there is no official support for Map, which is why I created my own module (currently in dev).
here's my VSCode extension to help with authoring TextMate grammars https://marketplace.visualstudio.com/items?itemName=RedCMD.tmlanguage-syntax-highlighter
