chumsky icon indicating copy to clipboard operation
chumsky copied to clipboard

Add support for custom merging span contexts

Open CraftSpider opened this issue 2 years ago • 3 comments

Currently, when trying to map with span, the parser will copy the context from the end-of-input. This makes preserving certain aside data, such as source ranges, difficult. Even with just a file ID, this may cause issues with expanded macro tokens. It would be nice to have some method on the Span trait to support spans 'merging' their contexts, something like fn merge(start: &Self::Context, end: &Self::Context, eoi: Self::Context) -> Self::Context which defaults to just returning eoi.

CraftSpider avatar Apr 20 '22 17:04 CraftSpider

Alternative solutions I've thought of:

  • Assign tokens keys and store source spans in a side table before parsing
  • Store a map of stream ranges to source ranges
  • Maybe you can just use the source range instead for chumsky's range? This would make indexing into your stream a pain

The biggest issue is that most solutions require more global state, which I'm trying to avoid

CraftSpider avatar Apr 20 '22 17:04 CraftSpider

Still an issue, but probably not enough to block zero-copy.

zesterer avatar Feb 20 '23 22:02 zesterer

A related issue is that of spans currently being 'cannibalised' out of existing tokens, thereby assuming that the input spans were continuous when in reality there's no promise of that being the case. Finding a solution to this issue seems difficult, though.

zesterer avatar Mar 05 '23 17:03 zesterer