streamly icon indicating copy to clipboard operation
streamly copied to clipboard

distribute scan combinators with statically generated scan

Open harendra-kumar opened this issue 8 months ago • 1 comments

We have combinators with monadically generated scans which is very powerful but can be inconvenient to use when we just need some static scans. Currently we have:

parDistributeScan :: MonadAsync m => (Config -> Config) -> m [Scanl m a b] -> Stream m a -> Stream m [b] 
parDemuxScan :: (MonadAsync m, Ord k) => (Config -> Config) -> (a -> k) -> (k -> m (Scanl m a b)) -> Stream m a -> Stream m [(k, b)] 

In the monadic versions if we want one-shot scans we have to use a destructive read so that the scans are deleted once they are used by the consumer. This is inconvenient for simple cases where we just need a static list of scans.

We can change the existing API names to have an M suffix and use these names for simpler non-monadic scan generation function.

harendra-kumar avatar May 14 '25 17:05 harendra-kumar

Adding more context: Another API suggested was,

parDistributeScan :: MonadAsync m =>
    (Config -> Config) -> Stream m [Scanl m a b] -> Stream m a -> Stream m [b]

But I've benchmarked this, and it is comparatively inefficient.

adithyaov avatar Jun 11 '25 12:06 adithyaov