Switching lexers

[ronniec95]

Is there a way to switch lexers? Standard case of lexing a buffer using a primary lexer and then a separate function to parse quoted strings, then switch back?

Is there an easy way to get the current position of the lexer and move it along while I take over?

[maciejhirsz]

Maybe. There is no way to do context switching on the Lexer atm, but you can do a lot with function callbacks. Have a look at this piece of code from Lunarity, it handleds the block comments (/* ... */) by exposing the internals of the Lexer and basically doing the work manually.

On a sidenote, quoted strings you can already do with regex. If you want the lexer to produce an unescaped String out of the literal, you can add a #[callback] there that reads through it and stores the result on some fields on Extras.

[jeertmans]

Closing this a too old, feel free to re-open if needed.