Make `fromAction` and `fromActionStep` able to pass a value from an action to the next one

[giulioforesto]

Context

On servant-server, I must serve some data that I query from a store. The persistance interface returns me some Stream of resources, so actual queries to the store are done only when the stream is consumed.

Unsatisfactory solution

Fold the stream into a list and respond with that list in a normal Servant endpoint. This is unsatisfactory because by folding the stream, all resources are read from the persistance layer and accumulated in memory before being sent, which leads me to huge memory usage.

Objective

Pipe the resources directly from the Prelude.Streaming.Stream into a Servant.Types.SourceT, so that the persistance layer is queried only when the resource must be sent in the response stream, and thus avoid to accumulate many resources in memory.

To achieve this, I can generate a SourceIO with a fromAction or fromActionStep, that reads the next element of the stream.

Issue

Since all the actions run by fromAction are actually the same, I cannot consume the stream by accessing the next values.

Solution

Tweak fromActionStep to make it pass a value from an action to the next one:

fromActionStep' :: Functor m => (c -> m (Maybe (a,c))) -> c -> StepT m a
fromActionStep' action = loop where
  loop c = Effect $ step <$> action c
  step Nothing = Stop
  step (Just (x,t)) = Yield x $ loop t

And then call it on Prelude.Streaming.uncons.

Of course, Maybe (a,c) can be generalized to any type v, provided that we pass additionally some stop :: v -> Bool, value :: v -> a and rest :: v -> v functions.

Question

Was there an easier out-of-the-box solution I did not see?

Otherwise, is it worth adding such fromActionStepWithConsumable method to the library?

[alpmestan]

@giulioforesto Hello! Wouldn't StreamResponse from this package allow you to seamlessly plug your existing stream into servant's server machinery?

I can see there: type ServerT (StreamResponse method status contentTypes :: *) m = m (Stream (Of ByteString) (ResourceT IO) ())

which means that if your endpoint runs in some monad YourMonad, the return type of your handler would have to be YourMonad (Stream (Of ByteString) (ResourceT IO) ()).

Regarding the implementation, you can see how the plumbing is done in https://hackage.haskell.org/package/servant-streaming-server-0.3.0.0/docs/src/Servant.Streaming.Server.Internal.html (bottom of that module)

[giulioforesto]

Hi! Thank you for your answer!

I did not know about that package. It could indeed work. However, I saw that it's no longer maintained (notably, because it is supposed to have been replaced by Servant.API.Stream by the way), and it's not on Stackage either. Since I'm working on professional project, I cannot rely on deprecated libraries, so in any case it would need some rework. Besides, as my workaround with fromActionStep' is not so complicated, it is maybe easier to keep this solution.

It's just that I was surprised that there was no way to pass a value to the action of fromActionStep, to be able to consume something, like in my case Stream, or possibly any other data structure.

More generally, it would be nice to have the possibility to easily create a SourceT m from something else than a List (see source), notably from structures that cannot be converted to a List (without loosing interesting properties, like in my case the Stream).

[alpmestan]

Fair enough, it seems reasonable to focus on the more recent streaming infrastructure we got.

I think the existing functions are mostly about supporting whatever was needed at the time, and could indeed be expanded to facilitate the life of users who end up relying on this by covering more scenarios, like that unfold-ish function you've been describing. All in all, while it's possible to implement something like this in your codebase by writing this function yourself in terms of StepT/SourceT constructors directly, I personally wouldn't be against adding more generic utility functions like yours if you were to put together a patch. :-)