streamly icon indicating copy to clipboard operation
streamly copied to clipboard

Parser `Step` representation: make ParserK and ParserD consistent

Open adithyaov opened this issue 2 years ago • 6 comments
trafficstars

In ParserD the Int in step indicates the amount of elements to backtrack.

  • Partial 3 a means go back by 3 elements before running the driver again.
  • Partial 0 a run the driver without going back.

In ParserK the Int in step indicates the relative stream position to be in. Partial (+3) a means go forward by 3 elements and run the driver. (Unimplemented) Partial 1 a means go forward by 3 elements and run the driver. Partial 0 a means run the driver over the current element again. Partial (-3) a means go back by 3 elements and run the driver.

We need to unify the representation. The question is which representation makes more sense.

ParserK representation is much better as it represents even the forward seek as well. Even if not for forward seek, it's more expressive in the sense that there is no implicit meaning of driver going forward. We explicitly control it by saying Partial 1.

adithyaov avatar Jul 14 '23 15:07 adithyaov

Let's change this in 0.11 to adopt the ParserK implementation in Parser as well. It is intuitive, it is required in ParserK for chunk parsing, it will enable chunk parsing in Parser as well, we can also use it for SIMD implementation for folds/parsers.

Need to change all existing parsers. Users who are using the Parser constructor to create custom parsers will be affected by the change. Need to update the changelog accordingly. The change required is - in Partial n, Continue n, Done n n should be changed to 1-n.

harendra-kumar avatar Jan 28 '24 09:01 harendra-kumar

We can deprecate the internal parser module, anyone using custom parsers would be using the internal module because Parser constructor is not exposed from the released module.

harendra-kumar avatar Jul 20 '24 23:07 harendra-kumar

See also #2459 .

harendra-kumar avatar Jul 20 '24 23:07 harendra-kumar