syntax
syntax copied to clipboard
Tokenizer: support returning an actual token from a handler
Currently a lex rule handler returns only a token type (potentially modifying yytext), E.g.
[`[0-9]+`, `yytext = yytext.slice(1); return 'NUMBER';`]
In this case "100" would result to:
{type: 'NUMBER', value: "00"}
We need also to support returning a full token (type-value pair) from a handler:
[`[0-9]+`, `return {type: 'NUMBER', value: 'Override!'}]
A use-case: matching spaces would return number of spaces, instead of the spaces themselves:
[`\\n( *)`, `return {type: 'INDENT', value: $1.length}]