Move tokenization to a web worker

[hediet]

Also see #77140

Plan of attack:

• [x] https://github.com/microsoft/vscode/issues/167288
• [ ] Make sure rule ids are stable by initializing injection rules eagerly
• [ ] Tokenize in the webworker
• [ ] Send over states by diffing against the state of the previous line. Send the delta as pop(n), push(state) instructions.

[IllusionMH]

At first I was thinking it's about .ts extension that is treated as Video Transport Stream (TS) 😆

[RedCMD]

lines inside the viewport are retokenized everytime the viewport changes (scrolling etc) this causes many lag spikes when scrolling over massively long lines that don't tokenize fully

this is different to when it was synchronous, as it would try to tokenize the line once and not try again until the user types

[hediet]

this is different to when it was synchronous, as it would try to tokenize the line once and not try again until the user types

I don't think this is true, it wouldn't tokenize lines only once when it used heuristically computed states to tokenize them.

[RedCMD]

me just scrolling normally at the same speed in both videos notice the lag spikes with async on and the flickering of colours at the bottom section of the document

test file is the builtin c extension c.tmLanguage.json file it is a single line of 72138 characters long vscode manages to tokenize roughly 40,000 chars before giving up roughly 45,500 chars when async is enabled

https://user-images.githubusercontent.com/33529441/225884890-b986e143-d78d-41fc-852b-835478aa2106.mp4

https://user-images.githubusercontent.com/33529441/225884948-29ffe8aa-419c-4472-9a5d-d40be11878d7.mp4

I don't think this is true, it wouldn't tokenize lines only once when it used heuristically computed states to tokenize them.

I don't understand is when multiple lines are present in the viewport?