Using tree-sitter for syntax highlight

[Sasanidas]

Tree sitter (https://tree-sitter.github.io/tree-sitter/) is a very popular library for parsing languages, it's more useful for non-lisp languages mainly for syntax highlight and some local refactoring. It also have a very interesting query-api(https://tree-sitter.github.io/tree-sitter/using-parsers#the-query-api).

Luckily for us, someone already did the bindings for the C API of tree-sitter https://github.com/death/cl-tree-sitter with an MIT license so we can probably use it.

[cxxxr]

Thank you for sharing! This is very interesting.

[CarsonEllsworth33]

I just started working on this feature. Not much progress made yet other than basic parsing through the REPL right now (no buffer parsing yet).

I am using death's cl-tree-sitter and it seems to work.
There is either an issue on my end or within death's parsing code as I can't load grammars that I have built locally unless I rename them from libtree-sitter-<language>.so to tree-sitter-<language>.so

[CarsonEllsworth33]

[CarsonEllsworth33]

[CarsonEllsworth33]

[CarsonEllsworth33]

This is the issue I mentioned in my first comment about having to rename the shared library.

[cianoc]

Did you make any progress on this?

[logoraz]

There is another set of CL bindings for libtree-sitter that you may want to try: https://github.com/garlic0x1/cl-treesitter

[cianoc]

Given the author - that's probably a pretty solid library.

[CarsonEllsworth33]

I approached trying to get tree-sitter into Lem when I was fresh to both Lem and common lisp. I got derailed a while back and am working on a few easier things currently to get better at CL and better learn Lem's internals, but still have it in a backlog of stuff I want to work on for Lem. Last time I checked the cl-tree-sitter binding by death was breaking due to a SAP null error (or something like it). So it may be worth a look into the bindings by garlic0x1 as mentioned above.

[CarsonEllsworth33]

Though in terms of some progress I did fork deaths binding to fix the few issues I mentioned above if people want to look at it.

[cianoc]

Garlic0x1 wrote a high level API on top of the low level CFFI stuff - so that seems worth trying.

I'm working on some code to bring which-key and Marginalia to Lem, but when I've done on that I may pivot to trying that library with something relatively easy, such as Rust-mode.

[nahuel]

Just to note that Emacs moved to treesitter, while Neovim, Helix and Zed already have it at their core, and VSCode is moving into it. Zed and VSCode in particular use the tree-sitter runtime compiled to WASM.

LSP and Treesitter support should be first-class on every modern editor, not only for syntax highlighting but also for structural editing.

[cianoc]

I don't think anyone disagrees with you, it's just that nobody has found the time to do it.

I've done some work playing around with the library I mentioned and it seems to work fine. My next step is to use it for Rust and see if I can create an alternate rust mode that uses treesitter, rather than an internal parser - and see how that goes.

I'm currently trying to fix an annoying bug with the prompt - when I'm done with that I'll pivot to see if I can at least get a proof of concept up and running.

[github-actions[bot]]

This issue has been inactive for {days} days. If there is no activity within the next 14 days, it will be automatically closed. Please add a comment if it is still relevant.

[github-actions[bot]]

Closing this issue due to inactivity. Please reopen or create a new issue if the problem persists.