forest icon indicating copy to clipboard operation
forest copied to clipboard

Analyze Forest heap allocation, find memory growth culprit

Open LesnyRumcajs opened this issue 3 years ago • 2 comments

Issue summary

forest process memory usage isn't bound. During sync to mainnet it grows to the point it gets OOMed. It is not necessarily a memory leak (though could be), it may be some constant pushing to a collection.

We need to:

  • [ ] analyze heap usage in forest using a memory profiler of choice (e.g. heaptrack),
  • [ ] find which data structure is the culprit for OOM,
  • [ ] provide a fix / workaround if feasible (if it ends up more complex we may need to discuss it), ~~document memory profiler usage for Forest as a markdown document in the repo.~~ (separate issue https://github.com/ChainSafe/forest/issues/1817)

Other information and links

LesnyRumcajs avatar Jun 21 '22 07:06 LesnyRumcajs

I suspect the culprit is RocksDB. It might be caching way too much information.

lemmih avatar Jun 21 '22 07:06 lemmih

It would be nice to have statistics on how much memory is used by RocksDB vs Forest.

lemmih avatar Jun 21 '22 07:06 lemmih