chumsky
chumsky copied to clipboard
Add support for custom merging span contexts
Currently, when trying to map with span, the parser will copy the context from the end-of-input. This makes preserving certain aside data, such as source ranges, difficult. Even with just a file ID, this may cause issues with expanded macro tokens. It would be nice to have some method on the Span
trait to support spans 'merging' their contexts, something like fn merge(start: &Self::Context, end: &Self::Context, eoi: Self::Context) -> Self::Context
which defaults to just returning eoi.
Alternative solutions I've thought of:
- Assign tokens keys and store source spans in a side table before parsing
- Store a map of stream ranges to source ranges
- Maybe you can just use the source range instead for chumsky's range? This would make indexing into your stream a pain
The biggest issue is that most solutions require more global state, which I'm trying to avoid
Still an issue, but probably not enough to block zero-copy
.
A related issue is that of spans currently being 'cannibalised' out of existing tokens, thereby assuming that the input spans were continuous when in reality there's no promise of that being the case. Finding a solution to this issue seems difficult, though.