facebook
Many empty memtables are created during MyRocks create index
Expected behavior
New MemTable should not be created if there are empty memtables.
Actual behavior
Many new (empty) MemTables are created during MyRocks create index. This is a benign bug, these empty MemTables are deleted on create index finishes.
Steps to reproduce the behavior
In MyRocks, create many indexes in parallel on existing large table while heavily writing to another table.
It seems these MemTables are created in IngestExternalFiles -> FlushMemTable -> SwitchMemtable.
We found this bug by our webview inspector for DB, gray are empty MemTables:
These empty MemTables get deleted:
@rockeet would like to get started in contributing to rocksdb. Shall I assign this to myself?
@rockeet would like to get started in contributing to rocksdb. Shall I assign this to myself?
Sure, thanks for your contribution!
@rockeet, I was going through the code. One thing I noticed was, isEmpty() check is absent before calling SwitchMemtable() in FlushMemTable() while rest all invocations of SwitchMemtable() has the check .(https://github.com/facebook/rocksdb/blob/1de69409805e196dae4daad4975f6f83080f8a7c/db/db_impl/db_impl_compaction_flush.cc#L2344)
Let me know if I am going in the right direction. Thanks!
@rockeet Can you tell me how can I reproduce this issue?
In MyRocks, create many indexes in parallel on existing large table while heavily writing to another table.
PR #12230 did not fix the issue, we applied the PR to our branch, the issue still exists:
The gray text is empty MemTable.
PR #12230 did not fix the issue, we applied the PR to our branch, the issue still exists:
The gray text is empty MemTable.
Are they still created in SwitchMemtable?
@rockeet I am trying to repro within RocksDB by writing a simple program. My intention is to check if Ingestion + write load is calling IngestExternalFiles -> FlushMemTable -> SwitchMemtable and also verify if at any time the memtable size is 0. I am having a hard time trying to reproing the issue. Am I missing something from trying to repro in RocksDB?
I am not familiar with the MyRocks setup and functionalities. It would be great if you could share me your setup or the steps that you followed. Thanks!
@rockeet I am trying to repro within RocksDB by writing a simple program. My intention is to check if Ingestion + write load is calling IngestExternalFiles -> FlushMemTable -> SwitchMemtable and also verify if at any time the memtable size is 0. I am having a hard time trying to reproing the issue. Am I missing something from trying to repro in RocksDB?
I am not familiar with the MyRocks setup and functionalities. It would be great if you could share me your setup or the steps that you followed. Thanks!
Your setup missed long lived iterators scanning the db while writting db.
@rockeet From my understanding, I thought SwitchMemtable is the only place where new Memtables are created. Since the issue is reproducible even with IsEmpty() check inside SwitchMemtable(), are there other places where Memtable can be created?
I added live iterators as well but couldn't get it to reproduce. The way I am checking is verifying if Memtable size is 0 in SwitchMemtable().. (ingesting multiple files with 1000+ keys and 10k concurrent writes and 1 iterator trying to read).
This is the sample test that I am writing. Would you recommend any changes to the test or the way I am verifying if there are empty memtables? https://github.com/pranavshenoy/speedb/commit/002c48c8931bf10144cb9dd47327ccc884d8c188
PR #12230 did not fix the issue, we applied the PR to our branch, the issue still exists:
The gray text is empty MemTable.
