Kampfkarren
Tokens returned out of order
full_moon version: 1.0.0-rc5
Tokens are returned out of order in function invocations or declarations with args, where the brackets are returned first, then the args themselves. Adding types to variables results in this behavior too, where colon delimiter is returned first, then the type and identifier.
I'm trying to use full_moon for syntax highlighting in a REPL using rustyline, and the formatting ends up like so.
Reproducible Example
Input:
local x: number ='s'
function f(arg: string) return "str" end
function x() return 12 end
math.random(3, 12)
Parsed Output:
local : number x='s'
function f() arg: string return "str" end
math.random()3, 12
-- This parses fine, since it does not have any args
function x() return 12 end
The code which parses the input using full_moon looks like this:
fn highlight<'l>(&self, line: &'l str, _: usize) -> Cow<'l, str> {
let ast = match full_moon::parse(&line) {
Ok(toks) => toks,
Err(_) => {
return Cow::Borrowed(line);
}
};
let formatted = ast
.tokens()
.map(|tok| {
let style = match tok.token_kind() {
TokenKind::Identifier => Style::new().italic().white(),
TokenKind::MultiLineComment | TokenKind::SingleLineComment => {
Style::new().dim()
}
TokenKind::Number => Style::new().bold().yellow(),
TokenKind::Shebang => Style::new().italic().green(),
TokenKind::StringLiteral => Style::new().green(),
TokenKind::Symbol => Style::new().magenta().bold(),
TokenKind::Eof | TokenKind::Whitespace | _ => Style::new(),
};
style.apply_to(tok).to_string()
})
.collect::<Vec<String>>()
.join("");
Cow::Owned(formatted)
}
I'm also trying to do syntax highlighting. Failing this function, what's the best way to iterate tokens in-order for this purpose? I overrode visit_token in a custom visitor class, but the results are ostensibly pre-order. Is overriding every leaf-type-node on the visitor struct my best bet?
