rnix-parser icon indicating copy to clipboard operation
rnix-parser copied to clipboard

fix: don't handle TOKEN_ERROR as trivial tokens.

Open hsjobeki opened this issue 2 years ago • 0 comments

Error Tokens should be seen as something other than trivial.

This means they are skipped during parsing but must be validated.

Instead, handle them and fail during parsing. The parser produces an error indicating which token would be expected at that position.

Summary & Motivation

Details are also in the mentioned issues below.

Backwards-incompatible changes

As far as I can tell, this doesn't break any backward compatibility. rnix should now fail, where it didn't fail on invalid expressions.

@tazjin @sternenseemann: Did I miss something? This shouldn't break anything, right?

Further context

To ensure correct error handling, I copied the error tests from the tokenizer_tests to the parser_tests. Now the parser should fail and handle the error tokens

The following issues are fixed:

#146 #148

hsjobeki avatar May 18 '23 20:05 hsjobeki