composewell
Operations involving multiple layers of containers
A typical example of multi-layer operations is concat with similar semantics as the list concat operation here. Streaming/Pipes/Machines concat a Foldable whereas Conduit concats a MonoFoldable which includes monomorphic containers as well. I am not clear about the pros and cons of both except for the high level fact that MonoFoldable includes more containers than Foldable. For example MonoFoldable includes Text and ByteString as well.
Any feedback on this is welcome.
Operations from base/Data.List
Involving two layers of containers, we need to decide on how to represent layers (Foldable/MonoFoldable/Streams) in general before attempting these:
Generating layers:
- [ ] subsequences
- [ ] permutations
- [ ] inits
- [ ] tails
Splitting:
- [ ] splitAt
- [ ] span
- [ ] break
- [ ] partition
Grouping:
- [ ] group
- [ ] groupBy
- [ ] groupWith from GHC.Exts
Transformation with layers:
- [ ] intercalate
- [ ] transpose
Eliminating layers:
- [ ] concat
- [ ] concatMap
Special case of Strings:
- [ ] lines
- [ ] words
- [ ] unlines
- [ ] unwords
Unzipping
- [ ] unzip
I am tending towards using streams of streams just like list concat is list of lists. Is streams of Foldable also required/useful if we have streams of streams?
To represent the nested streams in a streaming fashion we can use the FreeT transformation approach taken by pipes/pipes-bytestring.
FreeT transformations are a bit heavyweight tool, but they provide the streaming guarantee. However for regular day to day cases where the substreams are not too big and therefore accumulating is not a problem we can also provide the simpler buffering versions of these APIs. When nested streaming is really needed the FreeT versions of the APIs can be used instead.
Another way to do this might be to use another stream type with a monad instance that handles nesting of streams. The monad instance would peel a layer of streams and return the result for the next layer and so on. This will be more like how the "streaming" package monad instance works. Not sure whether this can fit elegantly with the whole model and other types.