In bookie 4.12.0, rocksdb(6.10.2) has memory leak?

[wolfstudy]

BUG REPORT

Describe the bug

Our 64GB machine uses 15GB of memory inside and outside the heap, but the machine kills the Bookie process directly because the system memory is not enough. The memory usage is all occupied by the Bookie process, so the only place to suspect is to call the sub-JNI part, and only rocksdb is left. The bookie version is 4.12.0, and the version of rocksdb is 6.10.2.

free -m output as follows:

              total        used        free      shared  buff/cache   available
Mem:          63088       58063         337         268        4688        4639
Swap:             0           0           0

dmesg -T output as follows:

And the JVM monitor as follows:

All configurations related to rocksdb are the default configuration and have not been modified:

#############################################################################
## DB Ledger storage configuration
#############################################################################

# These configs are used when the selected 'ledgerStorageClass' is
# org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage

# Size of Write Cache. Memory is allocated from JVM direct memory.
# Write cache is used to buffer entries before flushing into the entry log
# For good performance, it should be big enough to hold a substantial amount
# of entries in the flush interval
#  By default it will be allocated to 1/4th of the available direct memory
dbStorage_writeCacheMaxSizeMb=

# Size of Read cache. Memory is allocated from JVM direct memory.
# This read cache is pre-filled doing read-ahead whenever a cache miss happens
#  By default it will be allocated to 1/4th of the available direct memory
dbStorage_readAheadCacheMaxSizeMb=

# How many entries to pre-fill in cache after a read cache miss
dbStorage_readAheadCacheBatchSize=1000

## RocksDB specific configurations
## DbLedgerStorage uses RocksDB to store the indexes from
## (ledgerId, entryId) -> (entryLog, offset)

# Size of RocksDB block-cache. For best performance, this cache
# should be big enough to hold a significant portion of the index
# database which can reach ~2GB in some cases
# Default is to use 10% of the direct memory size
dbStorage_rocksDB_blockCacheSize=

# Other RocksDB specific tunables
dbStorage_rocksDB_writeBufferSizeMB=64
dbStorage_rocksDB_sstSizeInMB=64
dbStorage_rocksDB_blockSize=65536
dbStorage_rocksDB_bloomFilterBitsPerKey=10
dbStorage_rocksDB_numLevels=-1
dbStorage_rocksDB_numFilesInLevel0=4
dbStorage_rocksDB_maxSizeInLevel1MB=256

[dlg99]

there were multiple changes around RocksDb. Please try upgrading to BK 4.14.x (this should be easier than upgrade to 4.15).

[wolfstudy]

there were multiple changes around RocksDb. Please try upgrading to BK 4.14.x (this should be easier than upgrade to 4.15).

Thanks @dlg99 feedback, we are trying to upgrade Bookie version to 4.14.x. At the same time, I want to dump the memory in the heap to see if you can see what caused the memory leak. Is it because of the memory leak in RocksDB itself or because of a problem with the interface we call RocksDB JNI that caused the memory leak?

[hangc0276]

We tested RocksDB 6.16.4, the native memory leak issue still exists, and after upgrading the RocksDB to 6.29.4.1, this issue was fixed.