conduit icon indicating copy to clipboard operation
conduit copied to clipboard

Functor instance causes memory leak when used with Shake

Open matil019 opened this issue 8 months ago • 0 comments

I'm not sure whether this is the right place to ask, but I'd like to have your help.

When a conduit is run in an IO monad and it is lifted into Shake's Action monad (or Rules monad), and the conduit uses <$> directly or indirectly, a memory leak occurs.

This is my minimized example:

#!/usr/bin/env cabal
{- cabal:
build-depends:    base ^>=4.17.2.1
                , conduit ^>=1.3.5
                , shake ^>=0.19.8
default-language: GHC2021
ghc-options: -with-rtsopts=-M500M -threaded
-}
{- project:
with-compiler: ghc-9.4
-}

import Data.Conduit ((.|), runConduit)
import Data.Conduit.Combinators qualified as CC
import Development.Shake

main' :: IO ()
main' = do
  _ <- runConduit
     $ CC.yieldMany [(1 :: Integer)..]
    .| (id <$> CC.map id)
    .| CC.last
  pure ()

main :: IO ()
main = shakeArgs shakeOptions $ do
  want ["test"]

  phony "test" $ liftIO $ main'

The above program exits with "heap overflow" after a while thanks to the rtsopts.

If any of the following changes are made, the leak goes away:

  • main' replaces main to be called directly
  • (id <$> CC.map id) is removed or replaced with CC.map id

Notably, replacing CC.last with CC.lastDef undefined causes the memory leak, presumably because CC.lastDef is defined as lastDef a = fromMaybe a <$> last.

I couldn't minimize it further to see whether Shake is the true cause of the leak or not. I doubt it, but I don't know.

matil019 avatar Jun 14 '24 10:06 matil019