BrightID-Node
BrightID-Node copied to clipboard
BrightID
Improve memory usage of arangodb process
@TripleSpeeder reported that arangodb is using around half of 32GB dappnode memory.
It seems we have some sort of problem in garbage collection that is related to new transnational mechanism we are using in latest release to cache queries and speed up things when there are high number of queries that should run. Node one is also using around half of its 4G memory.
I think this is not a bug, but rather expected behaviour. After reading https://www.arangodb.com/docs/stable/tutorials-reduce-memory-footprint.html and especially https://www.arangodb.com/docs/stable/programs-arangod-rocksdb.html#write-buffers it looks like the RAM usage is mostly rocksdb write buffers.
By default rocksdb.total-write-buffer-size is calculated as (system RAM size - 2GiB) * 0.5. With 32GB on my machine this results in the observed 15GB RAM used.
I'm extending the DappNode package setup to allow setting additional parameters to running arangodb, allowing to modify the RAM usage behaviour.
Running with --rocksdb.total-write-buffer-size 4294967296 (4GB) does not show any difference in RAM usage. Attached image shows RAM usage before and after restart with this option.
I continue exploring other options.
Looks like the read cache is using up most of the space. Limiting read cache to 2Gigabyte with --rocksdb.block-cache-size 2000000000 --rocksdb.enforce-block-cache-size-limit true results in overal memory usage capped at ~4GB:
