effekt icon indicating copy to clipboard operation
effekt copied to clipboard

Adding a separate lexer

Open dvdvgt opened this issue 3 months ago • 0 comments

This PR aims to speed up the parsing phase of the compiler, as discussed in #412. Since we would want to have a separate lexer regardless of the possibility of switching to a different parsing library or tuning the current implementation for speed, this should be useful.

The lexer is handwritten, and I took care to make it reasonably fast (of course, I am happy to hear your feedback). The reason for a handwritten lexer are of course speed, however, at a loss of ease of making changes. Though, since I would not expect the keywords, syntax of string and number literals and those for comments to change anytime soon, this should be negligible. Also, we can offer better error messages for invalid syntax than with current implementation.

unterminated string example
def main() = {
  val x = "hello
  ()
} 
>>> Expected a statement

TODO

  • [X] Have a working lexer for Effekt's syntax
  • [X] Cover basics functionality with tests
  • [X] Add support for quoted strings / string interpolation: "result: ${ xs.map { x => x + 1 } }"
  • [ ] Integrate lexer into current parser by making the current parser implementation input agnostic -- first merge changes into kiama?

dvdvgt avatar Mar 26 '24 13:03 dvdvgt