lexer
lexer copied to clipboard
lookahead without modifying the lexer state?
I am using lexer along with your other project parse.
I am having trouble figuring out how I can lookahead on my input without modifying the lexer state.
In this example I wish for check-bar to be able to lookahead to see if the next token is "bar" and if it is to just ignore it without consuming the token and fail otherwise.
(define-lexer my-lexer (st)
("%s+" (values :next-token $$))
("foo" (values :foo $$))
("bar" (values :bar $$)))
(define-parser check-bar
"This parser will fail unless the next token is bar but will not consume the token."
(.let (state0 (.get))
(.do (.is :bar)
(.put state0))))
(define-parser foo-then-bar
(.let* ((foo (.is :foo))
(_ 'check-bar)
(bar (.is :bar)))
(.ret (list foo bar))))
(defun run-parser (string)
(with-lexer (lexer 'my-lexer string)
(with-token-reader (next-token lexer)
(parse 'foo-then-bar next-token))))
However the following repl output shows that check-bar consumes token from the lexer state:
PARSER> (run-parser "foo bar")
; Debugger entered on #<LEXER::LEX-ERROR {1004E95393}>
[1] PARSER>
; Evaluation aborted on #<LEXER::LEX-ERROR {1004E95393}>
PARSER> (run-parser "foo bar bar")
("foo" "bar")
T
How can I lookahead at my input without consuming any tokens?