streamly icon indicating copy to clipboard operation
streamly copied to clipboard

Change the Partial constructor of Fold type

Open harendra-kumar opened this issue 2 years ago • 1 comments

Currently we implement a scan using the extract function of folds, in a scan we can use "Partial s b" instead. Then we can leave extract to be called at the end to flush any internal state.

Earlier we did not introduce Partial s b because that required threading around the value b in the driver for the case when the stream ends without the fold returning Done. We do not need to thread around "b" in the driver because "extract" can still be used to extract it from "s" when needed if the stream ends without the fold finishing. The difference is that depending on the fold, "extract" may return a different result because we know that the stream has ended.

harendra-kumar avatar Mar 12 '22 08:03 harendra-kumar

Note: With #1514 and this the fold type would become pretty similar to the stream type.

harendra-kumar avatar Mar 12 '22 08:03 harendra-kumar

Using Partial s b to return the value did not work particularly well. Here are the comments from the code:

An alternative to using an "extract" function is to use "Partial s b" style
partial value so that we always emit the output value and there is no need
to extract. Then extract can be used for cleanup purposes. But in this case
in some cases we may need a "Continue" constructor where an output value is
not available, this was implicit earlier. Also, "b" should be lazy here so
that we do not always compute it even if we do not need it.
But keeping 'b' lazy does not let the fold optimize well. It leads to
significant regressions in the key-value folds.

harendra-kumar avatar Jul 13 '23 14:07 harendra-kumar