rocksdb icon indicating copy to clipboard operation
rocksdb copied to clipboard

How to access DB with pure memory?

Open ChangqingChen opened this issue 1 year ago • 6 comments

I need to access DB in pure memory to increase query latency.

ChangqingChen avatar Jan 02 '24 12:01 ChangqingChen

If mmap() and storing the data uncompressed are both OK then plain table may be of use - https://github.com/facebook/rocksdb/wiki/PlainTable-Format, https://github.com/facebook/rocksdb/wiki/RocksDB-In-Memory-Workload-Performance-Benchmarks

Otherwise you'd use block-based table and configure a large enough block cache to hold everything in memory.

Beyond that, what features to use depends on your workload/requirements

ajkr avatar Jan 02 '24 18:01 ajkr

If mmap() and storing the data uncompressed are both OK then plain table may be of use - https://github.com/facebook/rocksdb/wiki/PlainTable-Format, https://github.com/facebook/rocksdb/wiki/RocksDB-In-Memory-Workload-Performance-Benchmarks

Otherwise you'd use block-based table and configure a large enough block cache to hold everything in memory.

Beyond that, what features to use depends on your workload/requirements

plaintable is an SST file that supports pure memory. What do I do if I want to use pure memory functionality?

ChangqingChen avatar Jan 03 '24 03:01 ChangqingChen

If mmap() and storing the data uncompressed are both OK then plain table may be of use - https://github.com/facebook/rocksdb/wiki/PlainTable-Format, https://github.com/facebook/rocksdb/wiki/RocksDB-In-Memory-Workload-Performance-Benchmarks

Otherwise you'd use block-based table and configure a large enough block cache to hold everything in memory.

Beyond that, what features to use depends on your workload/requirements

rocksdb::Options options; options.table_factory.reset(NewPlainTableFactory()); options.prefix_extractor.reset(NewFixedPrefixTransform(8)); options.allow_mmap_reads = true;

Is that the way?

ChangqingChen avatar Jan 03 '24 10:01 ChangqingChen

Yes. Just make sure the prefix_extractor you are setting reflects what kind of range scans you will be doing (with plain table, range scan needs to be within a key prefix). If you aren't doing range scans you could try using NewNoopTransform().

ajkr avatar Jan 03 '24 16:01 ajkr

allow_mmap_reads

If I use PlainTable, do I need to set BlockCache?

ChangqingChen avatar Jan 05 '24 06:01 ChangqingChen

There is no way to set block cache for plain table users. That is usually OK since block cache's usual purpose is caching block-based table blocks. There won't be any such blocks for plain table users.

ajkr avatar Jan 05 '24 16:01 ajkr