chumsky icon indicating copy to clipboard operation
chumsky copied to clipboard

Allow `repeated` to yield an `Iterator` rather than collecting into a `Vec`

Open JohnDowson opened this issue 2 years ago • 1 comments

This would allow to skip Stream::from_iter dance when using Chumsky as lexer as well as parser.

JohnDowson avatar Mar 28 '22 09:03 JohnDowson

I don't think that yielding in general is possible (at least, without dropping a lot of performance on the floor and increasing the complexity of the crate massively). That said, I think it should be possible to have a trait IterParser<I, O>: Parser<I, O> extension trait with a method like parse_iter for specific types like Repeated and SeparatedBy.

Figuring out exactly how error propagation should work is still an issue too: an iterator producing (Vec<Self::Error>, Result<O, Self::Error>) is probably not particularly useful for common use-cases like producing an iterator of tokens, so we'd need to come up with some other 'side effect' channel where errors can be generated. Perhaps parse_iter collects errors internally and allows fetching them afterward? This might be more ergonomic, but also not well suited to some use-cases.

zesterer avatar Mar 28 '22 09:03 zesterer

This is eventually going to be possible with all the bells & whistles via #270, closing in anticipation.

zesterer avatar Feb 08 '23 22:02 zesterer