conduit icon indicating copy to clipboard operation
conduit copied to clipboard

Running a conduit in a function with an unrelated parameter twice leaks memory

Open matil019 opened this issue 8 months ago • 0 comments

Troubleshooting my own program, I stumbled upon another instance of memory leak.

When you call twice a function which ignores its parameter, and the function runs a conduit, it leaks memory.

This is my minimized example:

#!/usr/bin/env cabal
{- cabal:
build-depends:    base ^>=4.17.2.1
                , conduit ^>=1.3.5
default-language: GHC2021
ghc-options: -with-rtsopts=-M500M -threaded
-}
{- project:
with-compiler: ghc-9.4
-}

import Data.Conduit ((.|), runConduit)
import Data.Conduit.Combinators qualified as CC

main :: IO ()
main = do
  _ <- go ()
  _ <- go ()
  pure ()
  where
  go _ = runConduit
     $ CC.yieldMany [(1 :: Integer)..]
    .| CC.last

The above program exits with "Heap exhausted;" after a while thanks to the rtsopts.

Here is what I observed so far:

  • Not only CC.lastDef but also CC.last causes memory leak (unlike #510)
  • Moving go to the top level still causes memory leak
  • Using the parameter but irrelevantly with the conduit still causes memory leak (see below)
  • Replacing [(1 :: Integer)..] with iterate' succ (1 :: Integer) still causes memory leak
  • Calling go only once does not leak memory
  • Removing the paremeter altogether does not leak memory
  • Using the parameter as part of CC.yieldMany does not leak memory (see below)

Just evaluating the parameter but irrelevantly with the conduit still causes the memory leak:

-- Leaks memory, aborts
main :: IO ()
main = do
  _ <- go ()
  _ <- go ()
  pure ()
  where
  go x = do
    print x
    runConduit
       $ CC.yieldMany [(1::Integer)..]
      .| CC.last

But using the parameter for CC.yieldMany prevents memory leak:

-- Doesn't leak memory, loops forever
main :: IO ()
main = do
  let i = 1 :: Integer
  _ <- go i
  _ <- go i
  pure ()
  where
  go i = runConduit
     $ CC.yieldMany [i..]
    .| CC.last

I have completely no idea what is going on. Please let me know if I misunderstand the evaluation strategy of Haskell.

matil019 avatar Jun 14 '24 14:06 matil019