memory leak

[gelisam]

Even a simple pure () uses more and more memory (on ghc-8.6.5, about 4 Kb per loop):

import Control.Monad
import GHC.Stats
import Language.Haskell.Interpreter
import System.Mem

printMemoryConsumption :: IO ()
printMemoryConsumption = do
  performGC
  live_bytes <- gcdetails_live_bytes . gc <$> getRTSStats
  putStrLn $ show live_bytes ++ " bytes"

main :: IO ()
main = forever $ do
  runInterpreter $ pure ()
  printMemoryConsumption

[gelisam]

The good news is that the size of the leak does not appear to be significantly bigger if we interpret a computation which allocates a lot of memory.

[gelisam]

It gets bigger if we call eval often though.

[ghost]

@gelisam Can this be used in production?

The problem I am tackling is:

I need an interpreter/compiler that can be run as a microservice. The microservice will execute a one liner lambda expression with provided inputs(from external users) and return the result, in some sandboxed way.

I was thinking of using Hint + HaXR for this, can you please advise if Hint(current or past version) can be used for this?

[KiaraGrouwstra]

hm, I may have run into this as well.

I'm using hint for neural program synthesis, so basically I've been calling it a lot.

[gelisam]

@gelisam Can this be used in production?

I don't like the question. This is an old, stable codebase which has been passed on from maintainer to maintainer for years. But I do not offer paid support, and as per the license, I offer no warranty of merchantability or fitness for a particular purpose. I am not aware of any significant flaw which would cause me to advise you against using it in production, but if you do and something breaks in production, it is your responsibility, not mine.

[ghost]

Thanks for clearing things, its nice of you to maintain this library. For now, we have decided to create our own Lisp dialect, which will do for our use case.

[gelisam]

I have isolated the source of the leak to the installSupportModule function: if I comment everything in initialize except it, the leak remains, whereas if I comment it and keep everything else, theleak disappears.

[gelisam]

I made some progress, but I now think I have to start over.

By continuing to comment out more and more code, I have reduced the example program to one which does not use hint anymore, only the ghc library:

import Control.Monad
import GHC.Stats
import System.Mem
import qualified GHC
import qualified GHC.Paths

printMemoryConsumption :: IO ()
printMemoryConsumption = do
  performGC
  live_bytes <- gcdetails_live_bytes . gc <$> getRTSStats
  putStrLn $ show live_bytes ++ " bytes"

main :: IO ()
main = do
  forever $ do
    GHC.runGhcT (Just GHC.Paths.libdir) $ do
      GHC.load (GHC.LoadUpTo (GHC.mkModuleName "invalid module name"))
    printMemoryConsumption

So, the next step is to report this space leak upstream to the ghc team, right? Well, not so fast. The ghc team knows that ghc has space leaks, and they are focusing their efforts on their largest leaks. Is the one we found a large leak? Let's see... no, only 64 bytes per loop. Since the original program was leaking 4 kb per loop, the main source of the leak is elsewhere.

[gelisam]

The amount of memory leaked each loop goes up when I add a bunch of redundant id calls to the phantom file, but it does not (in fact it even goes down??) when I add a bunch of comments. It thus isn't the string itself which is leaking that memory, it is ghc's parsed representation. so if that's the leak, then the problem isn't the usual space leak in which we're holding to some thunk or some data structure, it's either a leak in the ghc library, or hint's cleanPhantomModules is not calling the right ghc functions to tell the ghc library to release the module.

[gelisam]

Hmm, it also goes up when I make the module smaller; that pretty much invalidates my hypothesis. I think the longer and shorter modules are probably both invalid Haskell, my test harness isn't printing exceptions, and the exceptional path is leaking a lot more memory than the happy path.

[markusschlegel]

Are there any updates on this issue? I ran the experiment above with GHC 9.2 and Hint 0.9.0.6 and the leak is still present. It leaks about 800 bytes per iteration on my machine.

[gelisam]

This is not a priority for me at the moment. Would you like to take a look? Hint needs more volunteers!