marked icon indicating copy to clipboard operation
marked copied to clipboard

Different lexer behavior when creating lexer class object without options

Open t4cmyk opened this issue 2 weeks ago • 2 comments

Marked version: 16.4.2

Describe the bug

import { marked } from "marked";

const lexer = new marked.Lexer();
const tokensA = marked.Lexer.lex("Test");
const tokensB = lexer.lex("Test");

The values of tokensA and tokensB should be the same instead tokensA is

[
    {
        "type": "paragraph",
        "raw": "Test",
        "text": "Test",
        "tokens": [
            {
                "type": "text",
                "raw": "Test",
                "text": "Test",
                "escaped": false
            }
        ]
    }
]

as expected but tokensB is

[
    {
        "type": "paragraph",
        "raw": "Test",
        "text": "Test",
        "tokens": []
    }
]

The problem is probably related to the shared _defaults option object.

To Reproduce Steps to reproduce the behavior: Run the code above

Expected behavior I would expect both options to give the same result.

t4cmyk avatar Dec 04 '25 21:12 t4cmyk

The issue seems to be that this.tokenizer.lexer = this; is in the constructor and the tokenizer is shared between the two different instances of the lexer. So since you run .lex after constructing them both the tokenizer.lexer is the last one constructed.

The solution would be to use a single instance of lexer or different marked instances so the tokenizers are different instances.

import { Marked } from "marked";

const marked1 = new Marked();
const marked2 = new Marked();

const lexer = new marked1.Lexer(marked1.defaults);
const tokensA = marked2.Lexer.lex("Test", marked2.defaults);
const tokensB = lexer.lex("Test");

UziTech avatar Dec 05 '25 07:12 UziTech

If we wanted to fix this in marked it would probably be best to set this.tokenizer.lexer = this; in Lexer.blockTokens and Lexer.inlineTokens. We would also want to do the same thing for renderer.parser in the Parser class.

I would be ok with that if you want to make a PR.

UziTech avatar Dec 05 '25 08:12 UziTech