Reduce the number of write buffers for infrequently used column families

[git-hulk]

Search before asking

• [X] I had searched in the issues and found no similar issues.

Motivation

Currently, all column families are shared the same number of write buffers which is unnecessary for infrequently used column families including pubsub, propagte, and zset_score. We can keep them to a const value like 1 to reduce the memory usage in some scenarios.

Solution

No response

Are you willing to submit a PR?

• [ ] I'm willing to submit a PR!

[LiuYuHui]

Please assign it to me, thanks a lot :D

[git-hulk]

@LiuYuHui Thank you!

[git-hulk]

Hi @LiuYuHui Are you still working on this issue?

[LiuYuHui]

Hi @git-hulk, I have had no extra time bandwidth recently, please feel free to assign it to other people.

[git-hulk]

@LiuYuHui Thanks for your quick response. Welcome back while you get time.

[jjz921024]

Hi @git-hulk , May i question? How do we implement this feature?

1. For each column family add a config, specifying its number of write buffers. eg: pubsub.max_write_buffer_number、propagte.max_write_buffer_number、zset_score.max_write_buffer_number ...
2. Add a lite mode config, When this config is set to true, create a new lite_table_opts for infrequently used column families. like:

rocksdb::BlockBasedTableOptions lite_table_opts = InitTableOptions();
rocksdb::ColumnFamilyOptions lite_opts(subkey_opts);
lite_opts.write_buffer_size = 32 * MiB;
lite_opts.max_write_buffer_number = 0;

// replace subkey_opts with lite_opts 
column_families.emplace_back(kZSetScoreColumnFamilyName, lite_opts);
column_families.emplace_back(kPubSubColumnFamilyName, lite_opts);
column_families.emplace_back(kPropagateColumnFamilyName, lite_opts);
column_families.emplace_back(kStreamColumnFamilyName, lite_opts);```

[git-hulk]

@jjz921024 Yes, I'm good with this implementation. Would you like to submit a PR?

[jjz921024]

Yes, I want to try

[git-hulk]

@jjz921024 Assigned, thank you!