rocksdb
rocksdb copied to clipboard
Cache capacity is twice of the expected size
Expected behavior
I want the max shared cache capacity to 1gb including 150mb mem tables. To do that, I use this config (jvm) that is coming from kafka streams, that is the same as lot of other libs using rocksdb, even not in Java :
class BoundedMemoryRocksDBConfigSetter : RocksDBConfigSetter {
override fun setConfig(storeName: String, options: Options, configs: Map<String, Any>) {
options.updateTableFormatConfig {
setBlockCache(cache)
setCacheIndexAndFilterBlocks(true)
setCacheIndexAndFilterBlocksWithHighPriority(true)
setPinTopLevelIndexAndFilter(true)
setOptimizeFiltersForMemory(true)
}
options.setWriteBufferManager(writeBufferManager)
options.setMaxWriteBufferNumber(3) // kafka streams default
options.setCompactionStyle(CompactionStyle.UNIVERSAL) // kafka streams default
options.setCompressionType(CompressionType.LZ4_COMPRESSION)
options.setStatsPersistPeriodSec(0) // disable dumping stats, unable to have them locally btw
}
private fun Options.updateTableFormatConfig(block: BlockBasedTableConfig.() -> Unit) {
setTableFormatConfig((this.tableFormatConfig() as BlockBasedTableConfig).apply(block))
}
override fun close(storeName: String, options: Options) {
// nothing to clean
}
companion object {
val cache = LRUCache(
/* capacity = */ 1 * 1024 * 1024 * 1024,
/* numShardBits = */ -1,
/* strictCapacityLimit = */ false,
/* highPriPoolRatio = */ 0.1,
)
val writeBufferManager = WriteBufferManager(
/* bufferSizeBytes = */ 150 * 1024 * 1024,
/* cache = */ cache,
/* allowStall = */ true,
)
}
}
It's deployed inside a container in kubernetes.
But after that, the rocksdb cache capacity indicates the twice (2gb). And the cache usage is near to 2gb.
Actual behavior
Get capacity returns exactly twice of the configured size, e.g. Returns 2gb when 1gb configured, 600mb when 300mb configured, 400 when 200 configured....
Steps to reproduce the behavior
Just use this config and it should be the same ?