syntax icon indicating copy to clipboard operation
syntax copied to clipboard

Tokenizer: support returning an actual token from a handler

Open DmitrySoshnikov opened this issue 9 years ago • 0 comments

Currently a lex rule handler returns only a token type (potentially modifying yytext), E.g.

[`[0-9]+`,  `yytext = yytext.slice(1); return 'NUMBER';`]

In this case "100" would result to:

{type: 'NUMBER', value: "00"}

We need also to support returning a full token (type-value pair) from a handler:

[`[0-9]+`,  `return {type: 'NUMBER', value: 'Override!'}]

A use-case: matching spaces would return number of spaces, instead of the spaces themselves:

[`\\n( *)`,  `return {type: 'INDENT', value: $1.length}]

DmitrySoshnikov avatar Jan 04 '17 01:01 DmitrySoshnikov