rocksdb_force_flush_memtable_and_lzero_now has a race

[mdcallag]

The race is that there is no pause between flushing memtables (rocksdb_flush_all_memtables) and requesting compaction from L0 to L1. So if flushing memtables triggers some L0->L1 compaction then the attempt at a manual flush can fail. Code is here.

While this code is there to catch a race, or maybe this race, it didn't based on the debug printfs I added because s.InvalidArgument() was not true. The diff for the debug printfs is here.

The workaround is to do this in my SQL scripts. First I flush the memtable, then guess at how long it might take for flush + compaction to finish, then request L0->L1 compaction:

set global rocksdb_force_flush_memtable_now=1;
select sleep(30);
set global rocksdb_force_flush_memtable_and_lzero_now=1;

I added printfs to the current code to see what happens. This is without the workaround:

2022-07-27T13:16:18.821162-07:00 12 [Note] [MY-010926] [Server] Access denied for user 'UNKNOWN_MYSQL_USER'@'localhost' (using password: NO)
2022-07-27T13:17:25.426564-07:00 18 [Note] [MY-000000] [Server] RocksDB: Manual memtable and L0 flush.
2022-07-27T13:17:25.705741-07:00 18 [Note] [MY-000000] [Server] RocksDB: ff for a CF default
2022-07-27T13:17:25.705788-07:00 18 [Note] [MY-000000] [Server] ... attempt
2022-07-27T13:17:25.705852-07:00 18 [Note] [MY-000000] [Server] ... calling CompactFiles
2022-07-27T13:17:25.706149-07:00 18 [Note] [MY-000000] [Server] ... not OK: Operation aborted: Necessary compaction input file /000041.sst is currently being compacted.
2022-07-27T13:17:25.706163-07:00 18 [Note] [MY-000000] [Server] RocksDB: ff for a CF __system__
2022-07-27T13:17:25.706169-07:00 18 [Note] [MY-000000] [Server] ... attempt
2022-07-27T13:17:25.706186-07:00 18 [Note] [MY-000000] [Server] ... calling CompactFiles
2022-07-27T13:17:25.707572-07:00 18 [Note] [MY-000000] [Server] ... OK
2022-07-27T13:17:25.707593-07:00 18 [Note] [MY-000000] [Server] ... attempt
2022-07-27T13:17:25.707607-07:00 18 [Note] [MY-000000] [Server] ... no files
2022-07-27T13:17:25.707624-07:00 18 [Warning] [MY-050100] [Server] Global variable 

This is with the workaround:

2022-07-27T13:24:39.063946-07:00 36 [Note] [MY-000000] [Server] RocksDB: Manual memtable and L0 flush.
2022-07-27T13:24:39.066436-07:00 36 [Note] [MY-000000] [Server] RocksDB: ff for a CF default
2022-07-27T13:24:39.066460-07:00 36 [Note] [MY-000000] [Server] ... attempt
2022-07-27T13:24:39.066500-07:00 36 [Note] [MY-000000] [Server] ... calling CompactFiles
2022-07-27T13:24:40.125763-07:00 36 [Note] [MY-000000] [Server] ... OK
2022-07-27T13:24:40.125795-07:00 36 [Note] [MY-000000] [Server] ... attempt
2022-07-27T13:24:40.125817-07:00 36 [Note] [MY-000000] [Server] ... no files
2022-07-27T13:24:40.125824-07:00 36 [Note] [MY-000000] [Server] RocksDB: ff for a CF __system__
2022-07-27T13:24:40.125830-07:00 36 [Note] [MY-000000] [Server] ... attempt
2022-07-27T13:24:40.125842-07:00 36 [Note] [MY-000000] [Server] ... calling CompactFiles
2022-07-27T13:24:40.127863-07:00 36 [Note] [MY-000000] [Server] ... OK
2022-07-27T13:24:40.127882-07:00 36 [Note] [MY-000000] [Server] ... attempt
2022-07-27T13:24:40.127897-07:00 36 [Note] [MY-000000] [Server] ... no files

[mdcallag]

Filed a feature request for RocksDB to get more behavior provided by it