[Bug] Infinite loop during `colorize()` in `MonarchTokenizer.getLoadStatus` when nesting languages

[mlajtos]

Reproducible in vscode.dev or in VS Code Desktop?

• [x] Not reproducible in vscode.dev or VS Code Desktop

Reproducible in the monaco editor playground?

• [x] Not reproducible in the monaco editor playground

Monaco Editor Playground Link

No response

Monaco Editor Playground Code

Reproduction Steps

No response

Actual (Problematic) Behavior

No response

Expected Behavior

No response

Additional Context

Using colorize(), if Lang1 can embed Lang2, and Lang2 can embed Lang1, the MonarchTokenizer.getLoadStatus will result in an infinite loop, eventually exceeding stack limit.

Let say we have lang Foo which has doubly-quoted strings which embed Markdown code...

monaco.languages.setMonarchTokensProvider("Foo", {
    defaultToken: "",
    tokenizer: {
      root: [
        /* rules */
        [/"/, { token: "string", next: "@string", nextEmbedded: 'markdown' }],
      ],
      string: [
        [/[^"]+/, { token: "string" }],
        [/"/, { token: '@rematch', next: '@pop', nextEmbedded: '@pop' }],
      ],
})

Problematic example:

str = "
  \`\`\`Foo
    1 + 1
  \`\`\`
"

(escaped tripple-backticks because formatting on GitHub)

A string in Foo lang contains a tripple-backticked Markdown code block with Foo code. Trying to colorize() this will result in a infinite loop of MonarchTokenizer.getLoadStatus().