nimly icon indicating copy to clipboard operation
nimly copied to clipboard

Add Function Lexer Produce a Token for EOF

Open loloicci opened this issue 3 years ago • 0 comments

Add an option to NimlLexer to lexNext / lexIter provide a specified token when the lexer reaches EOF only once.

Example:

# lexer setup
lexer.ignoreIf = someProc
lexer.produceEOFToken(tokenForEOF) # if it is not given, reaching EOF produces no token.

# use lexer
...

related: #70

loloicci avatar Mar 06 '21 15:03 loloicci