WriteBufferManager can't limit the memory usage when I create many db instances

[GOGOYAO]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

I use write buffer manager in blobdb to limit the total memory usage. I create 1000 db instances and set the max write buffer usage to 30GB. The size of write_buffer is 16MB and min_write_buffer_number_to_merge is 2. My rocksdb is v7.2.0. I keep writing value of 100KB size into rocksdb and observe the memory usage. I set max_background_flushes to cpu cores * 0.8 and max_background_compactions to cpu cores * 0.2

Expected behavior

The total memory usage should be around 30GB.

Actual behavior

The total memory usage could be more than 100GB. I found log as follows, 25075 2022/08/11-16:54:48.369826 140438477006592 [db/db_impl/db_impl_write.cc:1549] Flushing column family with oldest memtable entry. Write buffers are using 149782925312 bytes out of a total of 21474836480.

Steps to reproduce the behavior

1. use write buffer manager with the limit of 30GB
2. create 1000 db instances
3. keep writing data into the db instances

[ltamasi]

Just to confirm: you are sharing a single WriteBufferManager object among all 1000 database instances, right?

[GOGOYAO]

Just to confirm: you are sharing a single WriteBufferManager object among all 1000 database instances, right?

yes. I have a global WriteBufferManager for all 1000 database instances

[GOGOYAO]

any ideas ?@ltamasi

Just to confirm: you are sharing a single WriteBufferManager object among all 1000 database instances, right?

[ltamasi]

Chances are flushes can't keep up with the incoming writes. You could try increasing the number of flush threads/jobs; or, alternatively, you could set the allow_stall option to make WriteBufferManager stall writes instead of allowing the memory limit to be exceeded. (See https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager for more details.)