conduit icon indicating copy to clipboard operation
conduit copied to clipboard

Support incrementally feeding into Conduits

Open alexbiehl opened this issue 1 year ago • 1 comments

At work we had a situation where we have a fairly complex sink implemented using Conduits:

uploadLogs :: ConduitT LogEntry output m ()

It so happened that the code that used uploadLogs required refactoring and we ended up having to implement a function

type Logger m = LogEntry -> m ()

to upload the logs. Of course we can't just reuse the uploadLogs conduit -- it handles chunking, compression, buffering, uploading. It would be quite handy and made Conduits more easily integrate in existing systems if there was a function

withIncrementalSink :: MonadUnliftIO m => Conduit input Void  m () -> ( (input -> m ()) -> m r ) -> m r
withIncrementalSink sink action = ... 

this allows action to feed inputs incrementally and once action returns we'd signal end-of-input to the sink and shut it down.

What do you think? This proposal clearly is not exactly Conduits main purpose but would pose an adapter that allows integrating Conduits more easily into all sorts of places in a modern Haskell codebase.

alexbiehl avatar Jun 19 '23 20:06 alexbiehl

I don't see a problem with adding such a helper function.

snoyberg avatar Jun 21 '23 02:06 snoyberg