unexplained sudden increase in memory usage

[zaidoon1]

I suddenly started seeing the memory usage increase for no apparent reason and it has not come down yet:

I have various metrics tracking memory usage, other things happening in rocksdb but none explain why the memory usage went up to the level it hit without coming back down:

I've also attached the LOG file (although I got the LOG a day after this issue started so not sure how useful):

LOG.txt

Any idea or suggestions to what can cause this?

[zaidoon1]

is there a formula for calculating max memory usage? My initial understanding is that if set_cache_index_and_filter_blocks is set to true, then total memory usage by rocksdb can be calculated as follows:

block cache size per cf + max_write_buffer_number per cf x write_buffer_size per cf

for example, say we have 3 cfs and each is configured with the following values: block cache size = 1.5 GB write_buffer_size = 256 MB max_write_buffer_number = 4

then total memory usage will be 3 x 1.5 GB + 3 x 4 x 256 MB = 7.5 GB

But it seems this formula is still missing something?

[ajkr]

What allocator are you using? We use jemalloc. If you are using jemalloc, can you set dump_malloc_stats?

[zaidoon1]

I'm using jemalloc, I'll look into dump_malloc_stats

[zaidoon1]

I may have misunderstood your question, in my LOG file i see:

  block_cache_options:
    capacity : 33554432
    num_shard_bits : 6
    strict_capacity_limit : 0
    memory_allocator : None
    high_pri_pool_ratio: 0.500
    low_pri_pool_ratio: 0.000

looks like I can set an allocator for block cache? My application that has rocksdb bundled in it is using jemalloc, but I don't pass a memory allocator when constructing the block cache. Is that an issue?

[ajkr]

looks like I can set an allocator for block cache? My application that has rocksdb bundled in it is using jemalloc, but I don't pass a memory allocator when constructing the block cache. Is that an issue?

Probably not an issue. memory_allocator is useful only if you want to customize allocations for block cache blocks. For example, adding instrumentation or requesting them from a special jemalloc arena.

[zaidoon1]

so i tried setting dump_malloc_stats but nothing shows up. Note that i'm doing this from rust so i'm using the c api to do this.

rust code:

            db_opts.enable_statistics();
            db_opts.set_log_level(LogLevel::Info);
            db_opts.set_dump_malloc_stats(true);
            
            ...

           openDB();

which translates to calling the following c functions:

rocksdb_options_enable_statistics
rocksdb_options_set_info_log_level
rocksdb_options_set_dump_malloc_stats

And per https://github.com/facebook/rocksdb/blob/d1386de632333e936b5bed63510c5d89d6cd18b4/db/db_impl/db_impl.cc#L1166-L1173

I expect to see a Malloc STATS section but don't see that.

Is there something else I need to call/set to get this to show up?

[zaidoon1]

correction, it turns out the rust crate by default doesn't build rocksdb with jemalloc and it gates it behind a feature flag so I actually wasn't using jemalloc for rocksdb part at least which explains why the stats were not printing. I'll enable jemalloc with rocksdb and see how that changes perf/memory utilization and will report back if that on its own resolves the issue.

[zaidoon1]

so even after enabling the feature flag, I still don't see the stats. Just to make sure, the rust library tries to get rocksdb to use jemalloc by setting:

config.define("WITH_JEMALLOC", "ON");

is WITH_JEMALLOC ON the correct flag?

[mdcallag]

In past tests I usually see RSS with glibc malloc being 2X or more larger than RSS with jemalloc. One example - https://smalldatum.blogspot.com/2018/04/myrocks-malloc-and-fragmentation-strong.html

RocksDB can be a stress test for allocator fragmentation and jemalloc does better at avoiding it.

[zaidoon1]

update, the rust bindings are the problem as they were not actually getting rocksdb to use jemalloc. After fixing them locally, I can actually see the malloc stats. So i'll push this to prod and see whether that fixes all my memory problems!

[zaidoon1]

a feature request that might be helpful is to include in the LOG the allocator being used by rocksdb to make sure that users don't think they are using something that they are not actually using.

[ajkr]

a feature request that might be helpful is to include in the LOG the allocator being used by rocksdb to make sure that users don't think they are using something that they are not actually using.

Agreed. We could log the value of HasJemalloc() here to start with: https://github.com/facebook/rocksdb/blob/2940acac0088384b9fd146ce2a2ba5e2a298d5d5/db/db_impl/db_impl.cc#L138-L151

Feel free to submit a PR if you are interested

[leeqx]

I suddenly started seeing the memory usage increase for no apparent reason and it has not come down yet:

I have various metrics tracking memory usage, other things happening in rocksdb but none explain why the memory usage went up to the level it hit without coming back down:

I've also attached the LOG file (although I got the LOG a day after this issue started so not sure how useful):

LOG.txt

Any idea or suggestions to what can cause this?

please shared me the monitor code of Rocksdb? I need to monitor rocksdb's memory usage, thanks.

[zaidoon1]

@leeqx the rss memory i got from container_memory_rss{app_name=~"service_name.service"}. and other rocksdb memory usage metrics i got using https://github.com/zaidoon1/rust-rocksdb/blob/3e4a0f632a8c0c2839c7d183725c53895110d907/src/perf.rs#L267

Also update, after actually enabling jemalloc, I can confirm the memory issue is definitely fixed

thanks for the help everyone!