logos
logos copied to clipboard
maciejhirsz
Create ridiculously fast Lexers
This shouldn't be too difficult to add support for. As an example, `\\u\{(?:10|[0-9])[0-9a-fA-F]{0,4}\}` would just expand to `\\u\{(?:10|[0-9])[0-9a-fA-F]?[0-9a-fA-F]?[0-9a-fA-F]?[0-9a-fA-F]?`. That is, expand to `n` copies of the repeated node and `m-n`...
A good suggestion [came up on reddit](https://www.reddit.com/r/rust/comments/g3pfaw/logos_011_is_out_iterators_callbacks_and_stateful/fnu70z0/). Could add an extra flag into the top `#[logos]` attribute that points to a path to a dot file that can be used...
It would be nice if the `\W` anchor was supported in `#[regex]` strings such that certain regular expressions could succeed if EOF is encountered early. For example: ```rust #[derive(Logos)] enum...
From reddit: Question: would you consider adding a traditional code generation API in addition to proc macro? The problem with proc macros is that if you build a lexer using...
In the end I am using the following construct for dispatching sublexer. ``` fn sublexer, Subtok>(skip: usize, lex: &mut Lexer) -> Vec where Subtok: Logos + PartialEq + Clone +...
Hi! First of all, thanks for writing this super cool crate and making Rust's ecosystem more robust. I'm the original developer of pest and was wondering what would be the...
Is there a way to switch lexers? Standard case of lexing a buffer using a primary lexer and then a separate function to parse quoted strings, then switch back? Is...
I have two questions regarding `Clone` and `Copy` for `Lexer` and `SpannedIter`: - Would it be possible to add `Copy` to `Lexer` if both the token and the extras are...
This adds a small amount of extra documentation to the crate level docs about how to get started using the crate.
```rust #![allow(warnings)] #[derive(logos::Logos, Debug, Clone, Copy, PartialEq, Eq)] pub enum Token { #[regex(r"([0123456789]|#_#)*#.#[0123456789](_|#_#)?")] Decimal, #[regex(r#"..*"#)] BareIdentifier, #[error] Error, } ``` If I did the minification correctly: - `_` can be...