Tokenization freezes with long line like "((a+a) (a+a) (a+a) (a+a) ...

[alexr00]

From @Maxpxt in https://github.com/microsoft/vscode/issues/89846

Details

What editor are you seeing the problem in? (e.g. Atom, Visual Studio Code, etc.) VS Coe What version of the editor are you using? VS Code version: Code 1.41.1 (26076a4, 2019-12-18T14:58:56.166Z) What color theme are you using? Not applicable

Repro

Create a file in plain text mode and fill it with: ((a+a) + ~30 times  (a+a), which looks like ((a+a) (a+a) (a+a) (a+a) ...

Change the file language to C#.

This causes VSCode to stop responding.

This does not happen with other languages like C++, F#, Java, JavaScript.

Interestingly, in the following slight variations of this, everything works fine.

( on line 1 + (a+a) (a+a) (a+a) (a+a) ... on line 2 ((a+a)(a+a)(a+a)(a+a) ... When the line is smaller (around 15 to 20 times  (a+a)), the the editor does not stop responding, but takes long to respond to any changes made to the file (like typing a letter). The larger the line, the longer it takes.