Typing open parenthesis forces auto expand of collapsed regions

[barryt2]

Typing open parenthesis forces auto expand of collapsed regions.

Environment data

• Language Server version: 2022.6.3 on VS Code
• OS and version: Windows 11
• Python version (& distribution if applicable, e.g. Anaconda): 3.8.3

Code Snippet

def my_truck(x, y):
    return [x, y]

def test_a(): 
    return "a"

def test_b():
    return "b"

Repro Steps

1. Collapse test_a and test_b functions
2. Type open parenthesis on line return [x, y] before the open square bracket producing this return ([x, y]

Expected behavior

Typing open parenthesis should not force all my collapsed regions to expand.

Actual behavior

Typing open parenthesis forces all my collapsed regions to expand.

[debonte]

Due to the parse error, the parse tree ends after [x, y]. This causes us to only return folding ranges up to that point, which is why the regions lower down in the file disappear/expand.

Maybe in a case like this we should merge any folding ranges from our last response that appear after the parse error (i.e. further down the file) into the new response? That could result in a lot of non-sensical folding nodes if the parse error persists while the user continues making edits though.

@erictraut, should the parser be able to recover in this scenario? Are open parens particularly problematic for parser recovery?

[erictraut]

Open parens are difficult for parser recovery because new lines are ignored (stripped out) by the tokenizer while within parentheses. This means the parser has very little information to go on. In this particular case, the next token the parser sees after the open paren is a def keyword token.

I've attempted to make adjustments to the parser to perform better parse recovery in this case. With my changes, it consumes fewer tokens when it discovers the parse error, but in a practical sense, the behavior is arguably worse because the def is no longer consumed by the error and is now parsed in a context that doesn't make sense (since it's seen as being part of the return statement). To make matters worse, new lines and indents/dedents are omitted from the token stream from that point onward because the tokenizer thinks its within a parenthesized context. That leads to a cascade of additional errors. My experimental change doesn't address the issue reported by the OP. Collapsed ranges are still expanded because the following def statements are no longer seen as function declarations by the parser.

I don't see a way to fix this in a general sense. I can think of ways to introduce a bunch of heuristics to handle very specific situations, but those would be fragile and would easily fail. They would also involve making the tokenizer aware of the grammar, which is highly undesirable. I don't recommend this approach.

This isn't an issue with most languages, but Python is somewhat unique in how it conditionally includes newlines and indents/dedents as tokens in the token stream. Most other languages treat all newlines and indents as whitespace and use curly braces as a way to identify code blocks.

@barryt2, I suspect that very few Pylance users run into this particular issue because VS Code by default always inserts a closing paren when you type an opening paren. I presume that you have this feature disabled? If so, perhaps you would consider enabling it to avoid this situation?