lexer icon indicating copy to clipboard operation
lexer copied to clipboard

lookahead without modifying the lexer state?

Open NicholasBHubbard opened this issue 3 years ago • 0 comments

I am using lexer along with your other project parse.

I am having trouble figuring out how I can lookahead on my input without modifying the lexer state.

In this example I wish for check-bar to be able to lookahead to see if the next token is "bar" and if it is to just ignore it without consuming the token and fail otherwise.

(define-lexer my-lexer (st)
  ("%s+" (values :next-token $$))
  ("foo" (values :foo $$))
  ("bar" (values :bar $$)))

(define-parser check-bar
  "This parser will fail unless the next token is bar but will not consume the token."
  (.let (state0 (.get))
    (.do (.is :bar)
         (.put state0))))

(define-parser foo-then-bar
  (.let* ((foo (.is :foo))
          (_ 'check-bar)
          (bar (.is :bar)))
    (.ret (list foo bar))))

(defun run-parser (string)
  (with-lexer (lexer 'my-lexer string)
    (with-token-reader (next-token lexer)
      (parse 'foo-then-bar next-token))))

However the following repl output shows that check-bar consumes token from the lexer state:

PARSER> (run-parser "foo bar")
; Debugger entered on #<LEXER::LEX-ERROR {1004E95393}>
[1] PARSER> 
; Evaluation aborted on #<LEXER::LEX-ERROR {1004E95393}>
PARSER> (run-parser "foo bar bar")
("foo" "bar")
T

How can I lookahead at my input without consuming any tokens?

NicholasBHubbard avatar Jan 16 '22 22:01 NicholasBHubbard