Support for region folds

[Cretezy]

GDScript supports the #region and #endregion comments designed for code organization. In the Godot script editor, these are folds.

If possible, this package should support these to be folded (through Neovim).

[PrestonKnopp]

I believe this could work by creating a $.region node that contains $._simple_statements and $._compound_statement. Giving $._statement a choice between $.region or the aforementioned statement nodes. The fold query for nvim would be (region) @fold.

Should this be baked into the grammar? This will change the structure of the syntax tree. I don't see any other fold queries in nvim-treesitter for a region concept.

[PrestonKnopp]

I looked more into defining a region rule. It requires integrating with the scanning behavior of indent, dedent, newline, and comment. Unfortunately, not as simple as inserting it into $._statement because:

1. $.comment is marked as an extra and both internally and externally lex'd. It can appear anywhere in the AST.
2. Newlines and indentation are scanned through comments. Comments reset indentation count because they can appear anywhere regardless of indentation level.

So, to sum up what makes this difficult is whitespace indentation and comment nodes allowed to appear anywhere.

This is absolutely doable. scanner.c would need to be reworked.

Additionally, this cannot be worked around be querying a range of sibling nodes: https://github.com/nvim-treesitter/nvim-treesitter/discussions/7848

[NathanLovato]

Would it be possible or fine to treat the region markers as special tokens/comment variants? Or are you limited to marking anything starting with # as a comment token/having it trigger parsing through the scanner.c module?

I'm just dabbling into tree-sitter so I hope you'll excuse the beginner question. I'd just like to gauge if this is something I could try contributing (without wasting your time).

Thank you very much for your work on this parser by the way! I maintain the Zed integration and thanks to you it was really easy to add syntax highlighting and support for code outline/quick symbol search and jump.

[PrestonKnopp]

@NathanLovato Hey, thanks for reaching out!

I'm just dabbling into tree-sitter so I hope you'll excuse the beginner question. I'd just like to gauge if this is something I could try contributing (without wasting your time).

No time would be wasted! A pair of fresh eyes would be greatly appreciated if you're willing to take a look.

Would it be possible or fine to treat the region markers as special tokens/comment variants? Or are you limited to marking anything starting with # as a comment token/having it trigger parsing through the scanner.c module?

As far as I can tell, it's not possible with the current grammar configuration. The internal lexer will override regions with comments when tokenizing the region name. I believe it's because a comment is marked extra and takes precedence. Assigning a high precedence to the name field has no effect.

If we remove comments from extra, the following example parses correctly. However, comments break when inserted anywhere that is not manually specified in the grammar.

Example

Rule: ```js region: ($) => seq( "#region", field("name", /.*/), $._newline, repeat($._statement), "#regionend" ) ```

GDScript:

#region
pass
#endregion

#region Name
pass
#endregion

AST:

region
  pass_statement
comment
pass_statement
comment

The literal terminal "#region" is parsed successfully. Whereas the region with a name is parsed as comments.

---

Thank you! And great work on the Zed integration! If you have any feedback or QoL ideas for the grammar "API", issues are very welcomed.

[NathanLovato]

Right, it makes sense! I would definitely like to take a look, just please note I might be particularly slow due to limited time off and the things I have to maintain.

---

Regarding the Zed integration, I just took over work by other contributors and added a handful of features, so it's the result of multiple contributions. I'm just making sure it's maintained and people who contribute will see their work tested and integrated.

At the moment, I lack experience to give you feedback on the parsed AST. I've mostly needed time to figuring out how to use tree-sitter in Zed because there wasn't much documentation on the integration process but otherwise everything needed was there in the AST, which was clear. It's been easy to filter tokens in queries thanks to the level of detail of the AST.

[PrestonKnopp]

This has been contributed by NathanLovato!