lexical
lexical copied to clipboard
feature: add experimental lexer compiler
Still experimenting with this but a rough first pass can be seen.
The idea here is that instead of using the LexicalBuilder
to build up a LexerInterface
at runtime, you can instead provide the name of an enum, name for a generated class and the path for the generated file and skip all of the reflection etc.
This could be especially useful for larger lexers that have a bunch of tokens and therefore require more runtime processing.
Right now, the class that gets generated is basically the same as the RuntimeLexer
class. Has the same methods, but all of the constructor logic is gone.
I think for this to be made even better, we should abstract out the methods that a LexerInterface
class might use, such as matching a specific pattern, checking for tokens, etc into a trait like LexerUtilities
and then the compiled class and RuntimeLexer
can use that, reducing duplication between the two.
There are also likely edge cases where a RegEx pattern contains a character that needs to be escaped before being inlined into the generated class (single quotes I think are a problem).