conduit
conduit copied to clipboard
Functor instance causes memory leak when used with Shake
I'm not sure whether this is the right place to ask, but I'd like to have your help.
When a conduit is run in an IO
monad and it is lifted into Shake's Action
monad (or Rules
monad), and the conduit uses <$>
directly or indirectly, a memory leak occurs.
This is my minimized example:
#!/usr/bin/env cabal
{- cabal:
build-depends: base ^>=4.17.2.1
, conduit ^>=1.3.5
, shake ^>=0.19.8
default-language: GHC2021
ghc-options: -with-rtsopts=-M500M -threaded
-}
{- project:
with-compiler: ghc-9.4
-}
import Data.Conduit ((.|), runConduit)
import Data.Conduit.Combinators qualified as CC
import Development.Shake
main' :: IO ()
main' = do
_ <- runConduit
$ CC.yieldMany [(1 :: Integer)..]
.| (id <$> CC.map id)
.| CC.last
pure ()
main :: IO ()
main = shakeArgs shakeOptions $ do
want ["test"]
phony "test" $ liftIO $ main'
The above program exits with "heap overflow" after a while thanks to the rtsopts.
If any of the following changes are made, the leak goes away:
-
main'
replacesmain
to be called directly -
(id <$> CC.map id)
is removed or replaced withCC.map id
Notably, replacing CC.last
with CC.lastDef undefined
causes the memory leak, presumably because CC.lastDef
is defined as lastDef a = fromMaybe a <$> last
.
I couldn't minimize it further to see whether Shake is the true cause of the leak or not. I doubt it, but I don't know.