marked icon indicating copy to clipboard operation
marked copied to clipboard

let walkTokens replace the current token with multiple

Open calculuschild opened this issue 1 month ago • 1 comments

Describe the feature Currently, walkTokens lets you edit individual token properties as you walk the tree. However, you can't inject new tokens (except as children if the current token has a "tokens" property). I would like to take, say, an HTML token, and break it apart into multiple pieces as their own tokens. For example:

<div>content</div> is parsed into:

[
{type:"html", block:true, raw:"<div>\ncontent\n</div>", pre:false, text:"<div>\ncontent\n</div>"}
]

Which I would like to split into:

[
  {type:"html", block:true, raw:"<div>\n\n", pre:false, text:"<div>\n\n"}
  {type:"text", raw:"content", text:"content", escaped:false}
  {type:"html", block:true, raw:"</div>", pre:false, text:"</div>"}
]

Why is this feature necessary? walkTokens is a great tool to tweak the token tree, particularly when you want to slightly change the behavior of a built-in token without reimplementing the entire tokenizer along with its regex. But it would be nice to have more control over the tree beyond just changing the existing tokens.

Describe alternatives you've considered Fully re-implementing the tokenizer for built-in tokens does work, because you can break tokens down as much as you want, but extensions can quickly fall out-of-sync with Marked.js bug fixes, and requires a lot more maintenance work.

calculuschild avatar Nov 20 '25 04:11 calculuschild

Could this be an extension? something that uses the processAllTokens hook to allow updating the tokens? similar to how I did the token position extension

UziTech avatar Nov 21 '25 01:11 UziTech