lexical icon indicating copy to clipboard operation
lexical copied to clipboard

feature: add experimental lexer compiler

Open ryangjchandler opened this issue 1 year ago • 0 comments

Still experimenting with this but a rough first pass can be seen.

The idea here is that instead of using the LexicalBuilder to build up a LexerInterface at runtime, you can instead provide the name of an enum, name for a generated class and the path for the generated file and skip all of the reflection etc.

This could be especially useful for larger lexers that have a bunch of tokens and therefore require more runtime processing.

Right now, the class that gets generated is basically the same as the RuntimeLexer class. Has the same methods, but all of the constructor logic is gone.

I think for this to be made even better, we should abstract out the methods that a LexerInterface class might use, such as matching a specific pattern, checking for tokens, etc into a trait like LexerUtilities and then the compiled class and RuntimeLexer can use that, reducing duplication between the two.

There are also likely edge cases where a RegEx pattern contains a character that needs to be escaped before being inlined into the generated class (single quotes I think are a problem).

ryangjchandler avatar Aug 10 '23 00:08 ryangjchandler