Different lexer behavior when creating lexer class object without options
Marked version: 16.4.2
Describe the bug
import { marked } from "marked";
const lexer = new marked.Lexer();
const tokensA = marked.Lexer.lex("Test");
const tokensB = lexer.lex("Test");
The values of tokensA and tokensB should be the same instead tokensA is
[
{
"type": "paragraph",
"raw": "Test",
"text": "Test",
"tokens": [
{
"type": "text",
"raw": "Test",
"text": "Test",
"escaped": false
}
]
}
]
as expected but tokensB is
[
{
"type": "paragraph",
"raw": "Test",
"text": "Test",
"tokens": []
}
]
The problem is probably related to the shared _defaults option object.
To Reproduce Steps to reproduce the behavior: Run the code above
Expected behavior I would expect both options to give the same result.
The issue seems to be that this.tokenizer.lexer = this; is in the constructor and the tokenizer is shared between the two different instances of the lexer. So since you run .lex after constructing them both the tokenizer.lexer is the last one constructed.
The solution would be to use a single instance of lexer or different marked instances so the tokenizers are different instances.
import { Marked } from "marked";
const marked1 = new Marked();
const marked2 = new Marked();
const lexer = new marked1.Lexer(marked1.defaults);
const tokensA = marked2.Lexer.lex("Test", marked2.defaults);
const tokensB = lexer.lex("Test");
If we wanted to fix this in marked it would probably be best to set this.tokenizer.lexer = this; in Lexer.blockTokens and Lexer.inlineTokens. We would also want to do the same thing for renderer.parser in the Parser class.
I would be ok with that if you want to make a PR.