Doc comments #2

[resolritter]

I suspect the main problem with https://github.com/tree-sitter/tree-sitter-rust/pull/99 is what is pointed out in https://github.com/tree-sitter/tree-sitter-rust/pull/99#discussion_r781614623. Here I tried to fix that problem and also add some recursion guard tests just in case. EDIT: Some other cases were not being handled properly in #99 so I've included them and also added more tests.

This is being submitted as Draft so that @maxbrunsfeld can run the "randomized tests". I don't know how to run them.

https://github.com/tree-sitter/tree-sitter-rust/pull/99#issuecomment-1009361245 might be relevant and invalidate this approach altogether.

[mkrasnitski]

@resolritter any updates on this? This (and also code highlighting in docstring examples) are places where rust.vim currently has better (or rather, any) highlighting.

[cormacrelf]

I'm having a look at this in the hope that we can inject a markdown parser in doc comments. This is obviously a great start, but from what I can tell it's not quite optimal for that scenario. I think we want:

• Multiple successive doc comments contained, as they are here, in one doc_comment node, so editors can deal with successive doc comments as a group and for faster highlighting.
• However even when coalesced, each /// line or /** */ block goes into a child node of doc_comment.
• Each child node has the leading \s*///\s* or \s*//!\s* or \s*\*\s* skipped as if it's whitespace using lexer->advance(true);.
• The */ line uses lexer->mark_end() to strip that as well.

That way, when you pass the contents of doc_comment to a injected markdown parser, it doesn't have all the comment symbol noise and mess up the markdown parser. It would be much nicer if the lexer could skip the start of each line using advance(true), but it appears that this skips all the previous lines whenever you call it, i.e. the TS lever API assumes whitespace is only skippable at the start and end of tokens.

[mkrasnitski]

Bumping, as it would be great to finally get some more activity on this. How would this idea interact with code examples? It'd be great if writing examples could be as seemless as writing normal rust code, where each subsequent line gets a /// auto-inserted. Having plugins like vim-closer/auto-pairs work in that context would also be amazing.

[archseer]

@mkrasnitski Maybe you could pick up the PR and finish the implementation?

@cormacrelf We've solved this for other grammars by using combined injections, @the-mikedavis explains it here https://github.com/helix-editor/helix/pull/4870#issue-1462503103

[mkrasnitski]

I'm just a casual observer in this case - I don't yet feel qualified enough to pick this up myself, as I have really no familiarity with tree sitter internals. This is just one place where I feel tree sitter is not at feature parity with the official rust.vim plugin, and so any forward progress gets me excited.

[cormacrelf]

The combined injections trick works. Thanks for the tip @archseer! Will PR separately, based on this PR (to handle the more-than-three-/ case, which Erlang doesn't have to worry about).

The only thing I can't do is get Neovim to use Rust highlights for fenced code blocks without an explicit language tag. Tangentially I think you could do that by adding a feature to nvim-treesitter where you can create an alias for the markdown parser, but with separate query files (that will most likely use ; inherits: markdown). It can currently do aliases but they don't get their own query files. This thin filetype would be called rustdoc, used just for injections.

[resolritter]

I will not go forward with this PR. Feel to continue the work in a new PR.