Invalid position of comment node after `user_command` when next line starts with a whitespace

[EtiamNullam]

I've noticed some of comments in my vimrc where I subscribe to plugins via vim-plug are not highlighted. I managed to find a minimal scenario where the problem is apparent.

Consider the listing below. If 2nd line starts with a whitespace (empty line is fine) then the comment in the 1st line will not be detected.

Plug 'x/y' "this wont appear as a comment node
 "if this line starts with a space

Above listing will result in this tree:

user_command [0, 0] - [0, 10]
  command_name [0, 0] - [0, 4]
  arguments [0, 5] - [0, 10]
    command_argument [0, 5] - [0, 10]
      string_literal [0, 5] - [0, 10]
comment [1, 1] - [1, 1]
comment [1, 1] - [1, 34]

As you can see there is 0-length node for comment (comment [1, 1] - [1, 1]) and its on wrong line. Instead a correct node in this scenario should be: comment [0, 11] - [0, 46].

Screenshot:

[vigoux]

Hi and thanks for the reproduction steps. I'll try to work on this ASAP and add this as a test case. Thanks!!

[vigoux]

So this seems to be pretty strange. When looking into the issue, it seems that the scanner.c is correctly lexing the comment, but that the issue happens later.

It is also really linked to the space before the comment.

[vigoux]

So, the origin of the problem is multiline strings.

For some reason though, tree-sitter does not really like the fact that we call mark_end it seems. When running the example with gdb, I don't see any problem in the execution. I might open an issue in tree-sitter itself in order to check that.

[EtiamNullam]

Thanks for looking into it :)

So it seems like a problem with tree-sitter itself? Make sure to link the issue if you open one there.

[vigoux]

I'll definitely do!