Highlight matching do-end pairs

[ymtszw]

From the original: https://github.com/JakeBecker/vscode-elixir-ls/issues/90

Is this considered after forked? Personally I'm looking forward to the feature, so let me raise an issue for tracking. (I may try working on the feature but need to grasp project structure first)

[axelson]

It would be great to have, PR's welcome :)

You'll probably want to look inside syntaxes/elixir.json: https://github.com/axelson/vscode-elixir-ls/blob/a4764a564247d7c0aa8cd7f56e30bea9cc072c0c/syntaxes/elixir.json

[ymtszw]

Progress so far: In language extension configuration, there's brackets definition for automatic highlighting of matching brackets and other tokens. It supports multi-character tokens so we can simply add do-end pair, and it PARTIALLY works:

diff --git a/elixir-language-configuration.json b/elixir-language-configuration.json
index b0b160f..9bbc60d 100644
--- a/elixir-language-configuration.json
+++ b/elixir-language-configuration.json
@@ -3,6 +3,7 @@
     "lineComment": "#"
   },
   "brackets": [
+    ["do", "end"],
     ["{", "}"],
     ["[", "]"],
     ["(", ")"]

Result:

However there are big shortcomings:

1. We also have fn-end pairs, which conflict in their closing tokens (end)
   • VSCode clearly confused by conflicting closing tokens. When I also added fn-end to the brackets definition, pairings are broken.
2. :do can also appear in macro definitions: defmacro foobar(do: block) do
3. do-end blocks have one-liner variants: foobar(), do: :something
   • These two problems may be solved if token definition accepts regexp, using negative look-behinds and look-aheads. BUT I doubt so.

BTW syntaxes/elixir.json are for static grammatical colorings. In this issue we need to "dynamically" find closest matching pairs based on current cursor location. With the above observation, I think we have to implement our own (elixir-specific) pair-highlighting feature (in server, or in client with typescript?) since standard configuration feature cannot fulfill our needs.

[chaodhib]

@ymtszw The feature that you mentioned at the end of your comment, it is called Semantic highlighting and it has released been recently to VS Code: https://github.com/Microsoft/vscode/issues/585

https://github.com/microsoft/vscode/wiki/Semantic-Highlighting-Overview

[axelson]

@ymtszw yeah, those are some serious issues with adding proper matching for do, fn, end. I wonder if we could use the Semantic highlighting that chaodhib mentions, specifically for just do, end.

[dustypomerleau]

My understanding of semantic highlighting is that—rather than using regex-based matching of tokens—it instead builds a syntax tree for the entire document, which is how it recognizes multiple levels of nesting. This is currently only implemented for TS and JS. Because it's still experimental, they haven't provided much documentation on how to do this for other languages (yet), but presumably they will eventually provide a clear API with docs.

Semantic highlighting defines a new set of scopes for colorizing code based on its location in the tree, but I believe it is still limited to foreground colors (see this open issue). Part of the confusion stems from using the word highlight to mean 2 different things (foreground colorizing and background highlighting). On initial load, semantic highlighting is slow (several seconds delay in larger files). After the initial tree is built, it should be faster, but only testing will show if the lag is prohibitive for this use case.

I think it's worth looking at the code that is being used for editor.wordHighlight, which does modify the background color (in this case for words matching the current cursor position). This seems like a more similar use case. It registers background colors for the highlighted words, uses the Position module to determine cursor position, and then leverages VS Code's DocumentHighlight editor mode to background-highlight the matches. See this file.

[fishtreesugar]

it's possible reference ruby's implementation? https://github.com/rubyide/vscode-ruby/blob/7b5b6025caa2ebc140192e9add094aa34e7cec2d/packages/vscode-ruby-client/src/providers/highlight.ts

[lukaszsamson]

I read through the comments in https://github.com/microsoft/vscode/issues/26121 and https://github.com/microsoft/vscode/issues/48332. Apparently VSCode is now better at matching overlapping bracket pairs. Adding this to the bracket config makes a pretty good approximation of what is needed for elixir without implementing a custom semantic highlighter

    ["do", "end"],
    ["fn", "end"],
    ["do:", " "],
    ["do:", "\n"]

[lukaszsamson]

Changes reverted due to https://github.com/elixir-lsp/vscode-elixir-ls/issues/344

[lukaszsamson]

I was able to make do-end bracket matching work correctly with VSCode unbalancedBracketScopes. Specifically I made sure that atoms like :do are marked as unbalanced. VSCode by default also treats strings and documentation textmate scopes as unbalanced. I needed to change how charlists are matched - they use to be support.function.variable.quoted.single.elixir (which was totally invalid). Now they are matched as string.quoted.single.elixir.