Tree-sitter

[fox0430]

[kevinmatthes]

@fox0430

Tree-Sitters usually have C, C++, and / or JavaScript bindings, at least those I saw so far. They are plain parsers without any syntax highlighting. The highlighting itself is controlled by configuration files which are named highlights.scm, by convention. The parsing results are post-processed using these configuration files, collectively known as Tree-Sitter Queries. A Tree-Sitter Query can handle multiple categories of tokens, such as keywords, identifiers and punctuation. Applications binding Tree-Sitters need to provide colouring schemes for each category. The most important example therefore is the Haskell Tree-Sitter which defines that many categories such that some editors run out of colours and begin to apply wrong schemes for special sub-categories of keywords.

The difficulty is that not all Tree-Sitters provide Queries for highlighting, for instance the YAML Tree-Sitter, or do not cover all essential categories, such as the C++ Tree-Sitter which does not highlight comments. Others violate against the convention for the default path of the highlighting Query, such as the Just Tree-Sitter.

In case of missing Queries, these can be usually taken from other editors using Tree-Sitters. Most editors I know applying Tree-Sitters are written in Rust and, hence, usually dual-licensed "Apache-2.0 OR MIT" which both work with Moe's GPL-3.0. So, replacing missing Tree-Sitter Queries is not that problem. But we need to define an interface on how to grab Queries which are placed in the wrong directory, such as for Just. Both Zee and Helix, use a per-language setting strategy. This allows for the specification of an "Emergency Override Path" for highlighting Queries placed in the wrong directory.

The named editors, Zee and Helix, provide default highlighting Queries. These are compiled into the executable in case that the respective Tree-Sitters do not provide them or the provided ones are faulty. We would need to do so, as well, in order to adjust the provided Queries for the capabilities of Moe -- especially regarding the actual amount of colour schemes in case of Haskell.

[kevinmatthes]

@fox0430

We should first of all focus on finding a Tree-Sitter for Nim. When switching to Tree-Sitters, we should rather keep the highlighting support for the language Moe is written in. As a fallback solution, we should not completely drop the current highlighting system but query it in case there should either be no Tree-Sitter for the respective language or the user explicitly wishes to use the current system for it. This could be a per-language setting option, named "classic parsing", "Nim parsing", "traditional parsing" or "legacy parsing", for instance.

As a starting point I would like to suggest to ask in the Nim main repository whether there is already an official Tree-Sitter known or planned. The namespace does not provide one publicly, yet, so asking for the creation of one might be a good idea. Okay?

[fox0430]

@kevinmatthes

Ok, Thank you for the details. This feature looks hard. so I don't plan to work to add Tree-Sitter soon. In my opinion, this is not relevant to Tree-Sitter, but I want to prioritize LSP as a new big feature.