streamly
streamly copied to clipboard
Parser `Step` representation: make ParserK and ParserD consistent
In ParserD the Int in step indicates the amount of elements to backtrack.
Partial 3 ameans go back by 3 elements before running the driver again.Partial 0 arun the driver without going back.
In ParserK the Int in step indicates the relative stream position to be in.
Partial (+3) a means go forward by 3 elements and run the driver. (Unimplemented)
Partial 1 a means go forward by 3 elements and run the driver.
Partial 0 a means run the driver over the current element again.
Partial (-3) a means go back by 3 elements and run the driver.
We need to unify the representation. The question is which representation makes more sense.
ParserK representation is much better as it represents even the forward seek as well.
Even if not for forward seek, it's more expressive in the sense that there is no implicit meaning of driver going forward. We explicitly control it by saying Partial 1.
Let's change this in 0.11 to adopt the ParserK implementation in Parser as well. It is intuitive, it is required in ParserK for chunk parsing, it will enable chunk parsing in Parser as well, we can also use it for SIMD implementation for folds/parsers.
Need to change all existing parsers. Users who are using the Parser constructor to create custom parsers will be affected by the change. Need to update the changelog accordingly. The change required is - in Partial n, Continue n, Done n n should be changed to 1-n.
We can deprecate the internal parser module, anyone using custom parsers would be using the internal module because Parser constructor is not exposed from the released module.
See also #2459 .