RocksDB v.8.3.2 Still reachable bytes and running background threads after shutdown.

[JorgeMarinoDev]

Hello,

We are currently working on a SIEM Tool project that is using RocksDB v8.3.2. Since this kind of programs must remain up and running for really long periods of time, in terms of weeks, even months, we have a strict policy regarding memory profiling on the RCs. valgrind_issue_memcheck_report.log

BEFORE adding the Module using RocksDB, we had this summary after doing a valgrind memcheck.

==56637== LEAK SUMMARY:
==56637==    definitely lost: 0 bytes in 0 blocks
==56637==    indirectly lost: 0 bytes in 0 blocks
==56637==      possibly lost: 0 bytes in 0 blocks
==56637==    still reachable: 0 bytes in 0 blocks
==56637==         suppressed: 0 bytes in 0 blocks

But, after adding RocksDB, a Single DB and a couple of Column Families, just by opening the DB and creating the CFs handlers, not performing any operation either, we had this summary after shutdown: (with 12,351 still reachable bytes).

==56637== LEAK SUMMARY:
==56637==    definitely lost: 0 bytes in 0 blocks
==56637==    indirectly lost: 0 bytes in 0 blocks
==56637==      possibly lost: 0 bytes in 0 blocks
==56637==    still reachable: 12,351 bytes in 158 blocks
==56637==         suppressed: 0 bytes in 0 blocks

Important Note : Several shutdown procedures were tried, following the official documentation and examples. It means:

• Destroying CFs handles before Close.
• Calling DB Close.
• Deleting the rocksdb::DB raw pointer.

Also checking the opStatus and trying Close twice as the source code suggests somewhere in a header comment.

Valgrind reports still reachable bytes related to the background threads:

• rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) (threadpool_imp.cc:350)
• rocksdb::ThreadLocalPtr::InitSingletons() (thread_local.cc:240)
• rocksdb::ThreadLocalPtr::StaticMeta::SetHandler(unsigned int, void ()(void)) (thread_local.cc:446)
• rocksdb::ThreadPoolImpl::Impl::Impl() (threadpool_imp.cc:176)

Even further, performing TSAN and checking the running threads, the BG worker threads don't get destroyed after closing rocksDB.

No operations were pending even. Just starting the environment.

issue : There is no way to free these bytes or close the bg threads.

Environment

RocksDB added with CPM

`CPMAddPackage(`
  NAME rocksdb
  GITHUB_REPOSITORY facebook/rocksdb
  GIT_TAG v8.3.2
  VERSION 8.3.2
  OPTIONS
    "WITH_GFLAGS OFF"
    "WITH_TESTS OFF"
    "WITH_BENCHMARK_TOOLS OFF"
    "WITH_TOOLS OFF"
    "WITH_FOLLY_DISTRIBUTED_MUTEX OFF"
    "USE_RTTI 1"
    "FAIL_ON_WARNINGS ON"
    "ROCKSDB_BUILD_SHARED FALSE"
    "CMAKE_DISABLE_FIND_PACKAGE_gtest TRUE"
)

Initialize Options

    m_rocksDBOptions = rocksdb::Options();
    m_rocksDBOptions.IncreaseParallelism();
    m_rocksDBOptions.OptimizeLevelStyleCompaction();
    m_rocksDBOptions.create_if_missing = true;

Open call

    ...
    std::vector<std::string> columnNames;
    rocksdb::DB::ListColumnFamilies(rocksdb::DBOptions(), dbNameFullPath, &columnNames);
    ...
    std::vector<rocksdb::ColumnFamilyDescriptor> cfDescriptors;
    std::vector<rocksdb::ColumnFamilyHandle*> cfHandles;
	rocksdb::DB::Open(m_rocksDBOptions, dbNameFullPath, cfDescriptors, &cfHandles, &rawRocksDBPtr);
	...

Shutdown

// For every CFHandle
pRocksDB->DestroyColumnFamilyHandle(ptr);
...
pRocksDB->DestroyColumnFamilyHandle(defaultCFptr);
...

// After

rawRocksDBPtr->Close(); // Makes no difference. Optional here.

delete rawRocksDBPtr;
rawRocksDBPtr = nullptr;

Expected behavior

==56637== LEAK SUMMARY:
==56637==    definitely lost: 0 bytes in 0 blocks
==56637==    indirectly lost: 0 bytes in 0 blocks
==56637==      possibly lost: 0 bytes in 0 blocks
==56637==    still reachable: 0 bytes in 0 blocks
==56637==         suppressed: 0 bytes in 0 blocks

• Background threads after shutdown : none

Actual behavior

==56637== LEAK SUMMARY:
==56637==    definitely lost: 0 bytes in 0 blocks
==56637==    indirectly lost: 0 bytes in 0 blocks
==56637==      possibly lost: 0 bytes in 0 blocks
==56637==    still reachable: 12,351 bytes in 158 blocks
==56637==         suppressed: 0 bytes in 0 blocks

• Background threads after shutdown : n for each CF open. Not zero.

Steps to reproduce the behavior

Environment was detailed above.

[git-hulk]

@JorgeMarinoDev Did you have a try at canceling all background threads first? Refer: https://github.com/apache/kvrocks/blob/unstable/src/storage/storage.cc#L76

[JorgeMarinoDev]

@JorgeMarinoDev Did you have a try at canceling all background threads first? Refer: https://github.com/apache/kvrocks/blob/unstable/src/storage/storage.cc#L76

Hello, we have tried that before as well. Still getting the same results. Background threads are not closed and memory still reachable related to threads. Nevertheless, this code is already executed implicitly when closing the DBImpl. Thanks for the suggestion.

==154048== LEAK SUMMARY:
==154048==    definitely lost: 0 bytes in 0 blocks
==154048==    indirectly lost: 0 bytes in 0 blocks
==154048==      possibly lost: 0 bytes in 0 blocks
==154048==    still reachable: 12,471 bytes in 163 blocks
==154048==         suppressed: 0 bytes in 0 blocks

Also attaching results with this explicit call:

valgrind_log_explicit_call.txt

@git-hulk

[ywave620]

There is a leak! But it is spurious.

In RocksDB, bg threads are resources shared among multiple DB. They won't be recycled until the end of the process. However, since destruction of them is a little tricky, the authors of RocksDB decide to NOT destruct them, and NOT free the memory they occuiped. After all, the process it about to exit, no need to care about memory reclaim.

See

https://github.com/facebook/rocksdb/blob/70bf5ef0934d46ec42870ee0869353fce086c12d/port/lang.h#L44-L45

RocksDB provides an option for perfectionists, if you build(you should not in production) RocksDB with the hint of address_sanitizer is going to be used, then it will do the destruction.