logos
logos copied to clipboard
maciejhirsz
Consider adding "generate Rust source code" mode
From reddit:
Question: would you consider adding a traditional code generation API in addition to proc macro?
The problem with proc macros is that if you build a lexer using Logos, then all of your consumers will have to compile Logos itself and associated proc-macro machinery. In contrast, if you literally generate the lexer’s code and commit it to the repo, lexer consumers wouldn’t need any additional dependencies.
Logging the issue so that I can be notified if/when it is fixed, and switch https://github.com/matklad/tom and other stuff to logos :)
A sketch of the API:
Ideally, we use a custom attribute to tell logos "do a real code-gen for this token enum", but custom attributes are unstable. So, I suggest using something like this instead: a token is defined in a foo.in.rs file, using the usual syntax:
#[derive(Logos, Debug, PartialEq)]
enum Token {
#[end]
End,
#[error]
Error,
#[token = "fast"]
Fast,
#[token = "."]
Period,
#[regex = "[a-zA-Z]+"]
Text,
}
logos-codegen export a function fn codegen(in: &Path, out: &Path) -> io::Result<()>, which reads the in file using syn, and generates to out both the generated lexer and the Token enum, with logos-specific attributes stripped. That is,
#[derive(Debug, PartialEq)]
enum Token {
End,
Error,
Fast,
Period,
Text,
}
impl logos::Logos for Token {
...
}
Note that codegen should avoid writing the file if the content on disk is equal to the generated one.
