Tree-Sitter / Collapse-Function for Python Grammar

[TristynAlxander]

I updated from v1.00 something to v1.130.1 to fix python syntax highlighting, which worked; however, now collapsing is now broken in ~~multiple non-trivial ways~~ [one slightly annoying way].

The first is a annoying but ultimately minor issue where python comments [in if blocks] confuse the program:

# This Collapses Fine
if(True):
  pass

# This Collapses Fine
if(True):
  pass
  # Comment


# This Collapses is Broken
if(True):
  # Only this comment Disappears
  pass # This remains
  # Comment - this remains

# This doesn't work either, but I'm not sure if it should anyway. 
if(True): pass
  pass
  # None of this collapses

# There is no issue in while blocks

# This Collapses Fine
while(True):
  pass

# This Collapses Fine
while(True):
  pass
  # Comment

# This Collapses is Fine
while(True):
  #  this comment Disappears
  pass # This remains Disappears
  # Comment - this remains Disappears

Edit: [Converting .md files to .txt files fixes my other problem, but read on to see how it scared the daylights out of me.] ~~The second issue ruins the entire editor for me. I store a lot of notes in markdown files -- like a decade worth of technical wet-lab protocols. It used to be that I could collapse any indentation in markdown files, now it only collapses on markdown notation. I really cannot understate just how bad this is for me personally. I will have to attempt to downgrade to prevent my day-to-day workflow collapsing on Monday when I can't navigate my own notes.~~

[savetheclocktower]

I'll look into this.

In the meantime, one surefire way to restore the old behavior for Markdown files is to force them to use the old-style grammar. (See the section called I want to go back to the old highlighting!)

[TristynAlxander]

No Worries, I just panicked a bit with the markdown. I figured it out, and I can easily write a script to change my suffixes. Thanks for such a rapid response though.

[TristynAlxander]

@savetheclocktower , might check the difference between if and while blocks. This issue doesn't appear in while blocks. I've just edited the post to include this information.

[savetheclocktower]

No Worries, I just panicked a bit with the markdown. I figured it out, and I can easily write a script to change my suffixes. Thanks for such a rapid response though.

Sure — if the file extension is unimportant to you and you're not using any of the other features of the Markdown grammar (particularly syntax highlighting)… then that works.

You can also force .md files to be plain text without changing their file extension.

[savetheclocktower]

As for the issue with code folding in Python: I'm not surprised! This might be a quick fix or it might be a bug in the underlying Tree-sitter parser. I'll let you know what I find out.

[savetheclocktower]

@TristynAlxander, the latest commit in #1377 should address the Python folding issue, and it'll likely go out in 1.131.0.

Thanks for the report!

[savetheclocktower]

As for the issue with Markdown and folds:

There are several ways that the editor can decide to fold the contents of your editor. The simplest — and the one with the most history, dating back to TextMate 1.0 — is entirely indentation-based.

Another strategy is language-specific and works even with unusual indentation styles: figure out what characters start and end logical blocks and use those as cues to start/end folds. In a C-style language this takes the form of patterns that describe { and }, typically as the last and first content on the line, respectively.

The modern Tree-sitter grammar system can fold even more accurately using the semantics of the parse tree. In most languages, logical blocks (like the consequence of an if) are parsed into a node called block, and it's dead-simple to tell the grammar it can fold any block node.

In Markdown’s case, semantic folding is probably the best strategy for most people. Markdown is not an indentation-heavy language, after all. The parser we use now smartly groups all content under a heading (until the end of the file or until the next heading of equal or greater importance) as its own node, so we can fold prose in a very logical fashion.

But this doesn't fit your workflow! And that's fair. I've just described three different folding strategies to you, and there's no technical reason why we couldn't just let you pick between them, overriding the default behavior for the grammar. That would allow you to keep the syntax highlighting and all the other great features of the newest Markdown grammar without the catastrophic consequences of not having indentation-based folds.

#895 tracks this feature. It's not high on my to-do list, but it's likely to get done eventually. In the meantime, I'm glad you've found a workaround that satisfies you!