purescript-pipes icon indicating copy to clipboard operation
purescript-pipes copied to clipboard

what's the story about parallelism?

Open safareli opened this issue 7 years ago • 2 comments

Let's say you have Producer of some values, and want to run them in parallel. is there something you can do now to do that, if not what needs to be added to support such use case.

safareli avatar Dec 07 '18 16:12 safareli

For example, this doesn't compile, as there is no instance of Monad for ParAff: hoist sequential (traverse (hoist parallel) xs))

safareli avatar Dec 07 '18 16:12 safareli

this worked for me:

mergeProducers :: forall t o a. Traversable t => t (Producer o Aff a) -> Producer o Aff (t a)
mergeProducers ps = do
  var <- lift AV.empty

  fib <- lift $ forkAff do
    let consumer i = lift (AV.put i var) *> pure unit
    x <- parTraverse (\p -> P.runEffectRec $ p //> consumer) ps
    AV.kill (error "finished") var
    pure x

  let
    loop = do
      res <- lift $ try (AV.take var)
      case res of
        Left err -> lift $ joinFiber fib
        Right e -> do
          yield e
          loop
  loop

safareli avatar Dec 09 '18 09:12 safareli