Memory Grows Without Limit when ran for a long time (rocksdb)

[insipx]

Especially as the RocksDB database is filled up when syncing with a node that is far ahead, substrate-archive will slowly increase in memory-usage until the primary instance (polkadot/kusama/etc) is restarted + substrate-archive is restarted.

I believe this to be related in some way to: https://github.com/facebook/rocksdb/issues/4112

Since substrate-archive is a secondary instance, max_open_files is set to -1 (no limit). It will keep all files open and force allocations. We should maybe try playing with the LRUCache options, MALLOC_ARENA_MAXand other rocksdb settings to try and mitigate this.

[dvdplm]

Interesting, and a bit concerning. I wonder if this has anything to do with the openethereum memory growth issues many users are reporting.

Out of curiosity, did you try running substrate with --database=paritydb?

EDIT: just saw #76 :)