feat: fallback to regex highlighting when treesitter is not available

[mikavilpas]

trafficstars

Issue

Documentation for completions can include markdown fenced code blocks that should be highlighted with the appropriate syntax. When a treesitter parser is not available for the language used in the codeblock, the code inside is not highlighted correctly.

Incorrect highlighting example:

This can be an issue when using https://github.com/mikavilpas/blink-ripgrep.nvim which displays ripgrep (rg) matches for all the files in the project. Many big projects include a large variety of programming languages, and the user may not have all the treesitter parsers installed for all the languages.

Solution

Right now, neovim comes bundled with a regex-based syntax highlighter that supports 743 languages. We can use this syntax highlighter as a fallback when a treesitter parser is not available for the language.

Improved highlighting example:

The solution is not perfect but an 80% solution. The entire documentation text is highlighted in the syntax of the first codeblock. This means the code will be displayed correctly, but any markdown syntax in the documentation will look weird.

The idea is to provide a "better than nothing" experience out of the box, but if the user wants better highlighting, they should install the parser(s) for the languages they are interested in 🙂

Related issue: https://github.com/Saghen/blink.cmp/issues/336

[mikavilpas]

Hmm I have been thinking that my use case might be quite special and it probably isn't the best at addressing regex based highlighting in blink in general.

Maybe there's another way to solve this. For example, to get colored highlights, I could provide a function in blink-ripgrep that blink could call when the documentation needs to be highlighted.

That way I could also add other decorations that are not supported by regex (or treesitter). For example, I would like to highlight the matched word with a different background color.

Let me know what you would prefer!

[saghen]

I think a function makes sense. It could look something like fn highlight_documentation(win, buf, ns). In #465, I'll move all the blink specific properties on the items into a blink field and we can revisit this

[mikavilpas]

Sounds good 👍🏻

I'll follow that PR, it looks like it enables a huge number of things that I like.

[mikavilpas]

This was implemented in #650 using a different approach.