rocksdb
rocksdb copied to clipboard
Published
20 hours ago •
facebook
facebook
Enable FIFO compaction to open a multi-level db
FIFO compaction can theoretically open a DB with any compaction style. However, the current code only allows FIFO compaction to open a DB with a single level.
This PR relaxes the limitation of FIFO compaction and allows it to open a DB with multiple levels. Below is the read / write / compaction behavior:
- The read behavior is untouched, and it works like a regular rocksdb instance.
- The write behavior is untouched as well. When a FIFO compacted DB is opened with multiple levels, all new files will still be in level 0, and no files will be moved to a different level.
- Compaction logic is extended. It will first identify the bottom-most non-empty level. Then, it will delete the oldest file in that level.
Test Plan
Added a new test to verify the migration from level to FIFO where the db has multiple levels. Extended existing test cases in db_test and db_basic_test to also verify all entries of a key after reopening the DB with FIFO compaction.
Hello, @ajkr. This PR extends FIFO compaction to work multi-level DB. I would like to collect your initial feedback about this.
Initially, I am thinking about extending the ldb tool to support converting between level and FIFO in place, but I later found out it's easier and less error-prone if we simply extend FIFO compaction to allow it to work with multi-level DB.
If the idea makes sense to you, I am more than happy to add additional test cases.
cc @siying
Added a new test that covers the migration from level to fifo where the db has multiple levels.
Hello, @ajkr. I need your input about this PR. Do you mind taking a look when you're more available? Thank you!
Hello, @ajkr and @siying. This PR extends FIFO compaction to open a multi-level DB. By default, FIFO only produces sst files in L0 (this part is the same as the current state.) When FIFO opens a multi-level DB, it will pick the sst file to delete from Lmax to L0.
Our use case needs this PR to enable smooth migration from level to FIFO. Otherwise it requires a full-compaction in order to achieve the migration.
Can you help review or assign someone to review the PR? Thank you!
Trying to understand the logic here. With the change, do we still compact all files to L0. In which case we place files in non-L0?
So FIFO still only generates SST files in L0 as usual. The difference here is that this PR makes FIFO able to "open" or "process" a db with multiple levels. The main change is to generalize FIFO's compaction picker so that it instead picks the last file in the linked-list from the bottommost non-empty level. In other words:
- If db has only L0, then FIFO does exactly the same thing as today.
- If db has multiple levels, then FIFO evits the sst file w/ the largest key from the bottom-most non-empty level (i.e. the bottom-most right-most sst file in the LSM tree).
Created https://github.com/facebook/rocksdb/pull/10726 on top of this PR that further improves the CompactionPicker logic for FIFO on non-L0 levels so that it will use the file creation time to pick the file to evict (compact).
Hello @siying, @ajkr. I've updated the PR and created https://github.com/facebook/rocksdb/pull/10726 which improves the multi-level FIFO further. You can view this one as the base support and view https://github.com/facebook/rocksdb/pull/10726 as the completion of the multi-level FIFO.
I know PRs like this require a deeper look, but we also hope these PRs will be included in the next RocksDB release. To speed up the discussion, I am happy to VC with you sometime next week to go over the PRs and answer possible questions you might have. Does it sound like a good plan?
Trying to understand the logic here. With the change, do we still compact all files to L0. In which case we place files in non-L0?
So FIFO still only generates SST files in L0 as usual. The difference here is that this PR makes FIFO able to "open" or "process" a db with multiple levels. The main change is to generalize FIFO's compaction picker so that it instead picks the last file in the linked-list from the bottommost non-empty level. In other words:
- If db has only L0, then FIFO does exactly the same thing as today.
- If db has multiple levels, then FIFO evits the sst file w/ the largest key from the bottom-most non-empty level (i.e. the bottom-most right-most sst file in the LSM tree).
It's hard for me to imagine how the Tree would look like. It looks like that files will be first pushed to deeper levels, and then up to L0. Then deeper level files are removed, and files are kept stored in L0. Eventually, non-L0 files are all removed and only L0 files left. But since files are always in L0, files will be kept pushed to L0. Is my understanding correct?
Thanks for the feedback, @siying. I think your understanding is correct.
It looks like that files will be first pushed to deeper levels, and then up to L0. FIFO never pushes files to non-L0. The use-case scenario is that the DB originally used Level compaction where files are in different levels, then we want to migrate the DB to FIFO.
In that case, FIFO will remove files from the deepest level, as the deepest level generally represents oldest files.
Then deeper level files are removed, and files are kept stored in L0. Eventually, non-L0 files are all removed and only L0 files left. But since files are always in L0, files will be kept pushed to L0. Is my understanding correct?
Yes, non-L0 files will be removed first, and files are only pushed to L0, so eventually it will converge to the regular FIFO compaction case where there will be only L0 files.
@siying, @jay-zhuang: Thanks for the discussion via VC. Here's the summary:
- Update the code comment mentioning the compaction picker logic for FIFO on non-L0 files are only for migration. FIFO will still only create files in L0 and never push them to non-L0 level.
- For non-L0 files, FIFOCompactionPicker will pick the left-most file instead of right-most, meaning files with the smallest key will be purged first (this also means we don't need #10726 that purges files based on file time stamp).
- Rebase.
- Update FIFOCompactionPicker::PickSizeCompaction to pick the left-most files in non-L0 case.
- Update FIFOCompactionPicker::PickCompactionToWarm to skip when there're non-L0 files.
- Update the test case to verify the FIFO deletion order.
- Add comment block for FIFOCompactionPicker::PickSizeCompaction mentioning non-L0 cases are only for migration.
@siying has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@siying: I am not able to reproduce the error locally. Does the following error message indicate hang or time out? Do you happen to have any additional logs available internally?
https://app.circleci.com/pipelines/github/facebook/rocksdb/19802/workflows/d4adb8c5-70be-4556-8e45-a4c5b6fa318f/jobs/506198
ETA: 0s Left: 1 AVG: 0.01s local:1/12631/100%/0.0s
UID PID PPID C STIME TTY TIME CMD
root 121022 8 0 21:40 pts/1 00:00:01 make -j32 check
root 155222 121022 0 21:45 pts/1 00:00:00 /usr/bin/bash -c if test "100%" != 1 \ && (
root 155228 155222 0 21:45 pts/1 00:00:00 make T= check_0
root 155555 155228 0 21:45 pts/1 00:00:00 /usr/bin/bash -c printf '%s\n' ''......\ 'To monitor subtest <duration,pass/fa
root 155563 155555 15 21:45 pts/1 00:00:42 perl build_tools/gnu_parallel -j100% --plain --joblog=LOG --eta --gnu --tmpdir=/
root 1828172 155563 0 21:46 pts/1 00:00:00 /usr/bin/bash -c t/run-db_test-DBTest.LevelReopenWithFIFO >& t/log-run-db_test-D
root 1828305 1828172 0 21:46 pts/1 00:00:00 /bin/sh t/run-db_test-DBTest.LevelReopenWithFIFO
root 1828405 1828305 4 21:46 pts/1 00:00:08 ./db_test --gtest_filter=DBTest.LevelReopenWithFIFO
root 2503144 155563 0 21:50 pts/1 00:00:00 ps -wf
make: *** [Makefile:1069: check] Terminated
make[1]: *** [Makefile:1013: check_0] Hangup
Too long with no output (exceeded 10m0s): context deadline exceeded
And, there's another type of error that I don't understand what the error is: https://app.circleci.com/pipelines/github/facebook/rocksdb/19802/workflows/12d70287-6f4d-45e6-be20-6cfbb2879f02/jobs/506168
#!/bin/bash -eo pipefail
tar -cvzf t.tar.gz t
tar: t: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
Exited with code exit status 2
CircleCI received exit code 2
@siying: I am not able to reproduce the error locally. Does the following error message indicate hang or time out? Do you happen to have any additional logs available internally? Yes I think it indicates hanging. I don't know how CircleCI permission is managed. Are you able to rerun a test with "Rerun Job with SSH" option? In that way, you should be able to SSH to the host and do some investigation.
Yes I think it indicates hanging. I don't know how CircleCI permission is managed. Are you able to rerun a test with "Rerun Job with SSH" option? In that way, you should be able to SSH to the host and do some investigation.
@siying: "Rerun Job with SSH' is disabled for me. Are you able to grant me permission?
@yhchiang-sol here is the stack trace of the hanging test:
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
* frame #0: 0x00007fff20311cde libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff20344e49 libsystem_pthread.dylib`_pthread_cond_wait + 1298
frame #2: 0x0000000107f66ea8 librocksdb.7.8.0.dylib`rocksdb::port::CondVar::Wait(this=0x00007fd451809270) at port_posix.cc:122:23
frame #3: 0x00000001075ad8d9 librocksdb.7.8.0.dylib`rocksdb::InstrumentedCondVar::WaitInternal(this=0x00007fd451809270) at instrumented_mutex.cc:69:9
frame #4: 0x00000001075ad884 librocksdb.7.8.0.dylib`rocksdb::InstrumentedCondVar::Wait(this=0x00007fd451809270) at instrumented_mutex.cc:62:3
frame #5: 0x0000000106f9104b librocksdb.7.8.0.dylib`rocksdb::DBImpl::WaitForCompact(this=0x00007fd451808800, wait_unscheduled=true) at db_impl_compaction_flush.cc:3852:12
frame #6: 0x0000000107061792 librocksdb.7.8.0.dylib`rocksdb::DBImpl::TEST_WaitForCompact(this=0x00007fd451808800, wait_unscheduled=true) at db_impl_debug.cc:181:10
frame #7: 0x000000010625f95f db_test`rocksdb::DBTest_LevelReopenWithFIFO_Test::TestBody(this=0x00007fd44f017e00) at db_test.cc:596:5
frame #8: 0x00000001064a92e4 db_test`void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(object=0x00007fd44f017e00, method=21 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00, location="the test body")(), char const*) at gtest-all.cc:3899:10
frame #9: 0x00000001064781bb db_test`void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(object=0x00007fd44f017e00, method=21 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00, location="the test body")(), char const*) at gtest-all.cc:3935:14
frame #10: 0x00000001064780f6 db_test`testing::Test::Run(this=0x00007fd44f017e00) at gtest-all.cc:3973:5
frame #11: 0x000000010647925a db_test`testing::TestInfo::Run(this=0x00007fd44ec36d30) at gtest-all.cc:4149:11
frame #12: 0x000000010647a094 db_test`testing::TestCase::Run(this=0x00007fd44ec36260) at gtest-all.cc:4267:28
frame #13: 0x00000001064857c8 db_test`testing::internal::UnitTestImpl::RunAllTests(this=0x00007fd44ec35f60) at gtest-all.cc:6633:43
frame #14: 0x00000001064ae294 db_test`bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(object=0x00007fd44ec35f60, method=20 55 48 06 01 00 00 00 00 00 00 00 00 00 00 00, location="auxiliary test code (environments or event listeners)")(), char const*) at gtest-all.cc:3899:10
frame #15: 0x00000001064852bb db_test`bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(object=0x00007fd44ec35f60, method=20 55 48 06 01 00 00 00 00 00 00 00 00 00 00 00, location="auxiliary test code (environments or event listeners)")(), char const*) at gtest-all.cc:3935:14
frame #16: 0x000000010648518d db_test`testing::UnitTest::Run(this=0x00000001065014a0) at gtest-all.cc:6242:10
frame #17: 0x000000010632d421 db_test`RUN_ALL_TESTS() at gtest.h:22110:46
frame #18: 0x000000010632d409 db_test`main(argc=1, argv=0x00007ffee99b20d0) at db_test.cc:7351:10
frame #19: 0x00007fff2035ff5d libdyld.dylib`start + 1
thread #2
frame #0: 0x00007fff20311cde libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff20344e49 libsystem_pthread.dylib`_pthread_cond_wait + 1298
frame #2: 0x00007fff202add72 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
frame #3: 0x0000000107b761e4 librocksdb.7.8.0.dylib`rocksdb::ThreadPoolImpl::Impl::BGThread(this=0x00007fd44ec50da0, thread_id=0) at threadpool_imp.cc:243:17
frame #4: 0x0000000107b76b18 librocksdb.7.8.0.dylib`rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(arg=0x00007fd44ec506f0) at threadpool_imp.cc:360:7
frame #5: 0x0000000107b7b525 librocksdb.7.8.0.dylib`decltype(__f=(0x00007fd44ec46048), __args=0x00007fd44ec46050)(void*)>(fp)(std::__1::forward<rocksdb::BGThreadMetadata*>(fp0))) std::__1::__invoke<void (*)(void*), rocksdb::BGThreadMetadata*>(void (*&&)(void*), rocksdb::BGThreadMetadata*&&) at type_traits:3747:1
frame #6: 0x0000000107b7b46e librocksdb.7.8.0.dylib`void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(void*), rocksdb::BGThreadMetadata*, 2ul>(__t=size=3, (null)=__tuple_indices<2> @ 0x000070000c34cf58)(void*), rocksdb::BGThreadMetadata*>&, std::__1::__tuple_indices<2ul>) at thread:280:5
frame #7: 0x0000000107b7ac56 librocksdb.7.8.0.dylib`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(void*), rocksdb::BGThreadMetadata*> >(__vp=0x00007fd44ec46040) at thread:291:5
frame #8: 0x00007fff203448fc libsystem_pthread.dylib`_pthread_start + 224
frame #9: 0x00007fff20340443 libsystem_pthread.dylib`thread_start + 15
thread #3
frame #0: 0x00007fff20311cde libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff20344e49 libsystem_pthread.dylib`_pthread_cond_wait + 1298
frame #2: 0x00007fff202add72 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 18
frame #3: 0x0000000107b761e4 librocksdb.7.8.0.dylib`rocksdb::ThreadPoolImpl::Impl::BGThread(this=0x00007fd44ec50e90, thread_id=0) at threadpool_imp.cc:243:17
frame #4: 0x0000000107b76b18 librocksdb.7.8.0.dylib`rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(arg=0x00007fd44ec47a90) at threadpool_imp.cc:360:7
frame #5: 0x0000000107b7b525 librocksdb.7.8.0.dylib`decltype(__f=(0x00007fd44ec522f8), __args=0x00007fd44ec52300)(void*)>(fp)(std::__1::forward<rocksdb::BGThreadMetadata*>(fp0))) std::__1::__invoke<void (*)(void*), rocksdb::BGThreadMetadata*>(void (*&&)(void*), rocksdb::BGThreadMetadata*&&) at type_traits:3747:1
frame #6: 0x0000000107b7b46e librocksdb.7.8.0.dylib`void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(void*), rocksdb::BGThreadMetadata*, 2ul>(__t=size=3, (null)=__tuple_indices<2> @ 0x000070000c3cff58)(void*), rocksdb::BGThreadMetadata*>&, std::__1::__tuple_indices<2ul>) at thread:280:5
frame #7: 0x0000000107b7ac56 librocksdb.7.8.0.dylib`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(void*), rocksdb::BGThreadMetadata*> >(__vp=0x00007fd44ec522f0) at thread:291:5
frame #8: 0x00007fff203448fc libsystem_pthread.dylib`_pthread_start + 224
frame #9: 0x00007fff20340443 libsystem_pthread.dylib`thread_start + 15
thread #4
frame #0: 0x00007fff20311cde libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff20344e49 libsystem_pthread.dylib`_pthread_cond_wait + 1298
frame #2: 0x0000000107f66f34 librocksdb.7.8.0.dylib`rocksdb::port::CondVar::TimedWait(this=0x0000000108128d30, abs_time_us=1665599157187521) at port_posix.cc:136:13
frame #3: 0x00000001075ada0c librocksdb.7.8.0.dylib`rocksdb::InstrumentedCondVar::TimedWaitInternal(this=0x0000000108128d30, abs_time_us=1665599157187521) at instrumented_mutex.cc:87:16
frame #4: 0x00000001075ad98c librocksdb.7.8.0.dylib`rocksdb::InstrumentedCondVar::TimedWait(this=0x0000000108128d30, abs_time_us=1665599157187521) at instrumented_mutex.cc:76:10
frame #5: 0x000000010721b33e librocksdb.7.8.0.dylib`rocksdb::Timer::Run(this=0x0000000108128cc8) at timer.h:253:19
frame #6: 0x000000010721c157 librocksdb.7.8.0.dylib`decltype(__f=0x00007fd44ed04718, __a0=0x00007fd44ed04728)).*fp()) std::__1::__invoke<void (rocksdb::Timer::*)(), rocksdb::Timer*, void>(void (rocksdb::Timer::*&&)(), rocksdb::Timer*&&) at type_traits:3688:1
frame #7: 0x000000010721c05e librocksdb.7.8.0.dylib`void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (rocksdb::Timer::*)(), rocksdb::Timer*, 2ul>(__t=size=3, (null)=__tuple_indices<2> @ 0x000070000c452f58)(), rocksdb::Timer*>&, std::__1::__tuple_indices<2ul>) at thread:280:5
frame #8: 0x000000010721b846 librocksdb.7.8.0.dylib`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (rocksdb::Timer::*)(), rocksdb::Timer*> >(__vp=0x00007fd44ed04710) at thread:291:5
frame #9: 0x00007fff203448fc libsystem_pthread.dylib`_pthread_start + 224
frame #10: 0x00007fff20340443 libsystem_pthread.dylib`thread_start + 15
and the latest LOG file looks like this:
static:~ distiller$ cat /private/tmp/rocksdbtest-502/db_test_8806_189362016023994854/LOG
2022/10/12-18:14:30.253664 4588350976 RocksDB version: 7.8.0
2022/10/12-18:14:30.254614 4588350976 Git sha 16fd058296c1ea53bb1ab8e2ec866609c871cf6f
2022/10/12-18:14:30.254625 4588350976 Compile date 2022-07-12 20:30:40
2022/10/12-18:14:30.254632 4588350976 DB SUMMARY
2022/10/12-18:14:30.254637 4588350976 DB Session ID: WSS1D7D499ICWHM3X7YP
2022/10/12-18:14:30.262835 4588350976 CURRENT file: CURRENT
2022/10/12-18:14:30.262850 4588350976 IDENTITY file: IDENTITY
2022/10/12-18:14:30.263068 4588350976 MANIFEST file: MANIFEST-002211 size: 527 Bytes
2022/10/12-18:14:30.263084 4588350976 SST files in /tmp/rocksdbtest-502/db_test_8806_189362016023994854 dir, Total Num: 5, files: 001305.sst 001349.sst 001393.sst 001437.sst 001481.sst
2022/10/12-18:14:30.263094 4588350976 Write Ahead Log file in /tmp/rocksdbtest-502/db_test_8806_189362016023994854: 001480.log size: 0 ; 001484.log size: 0 ; 001488.log size: 0 ; 001492.log size: 0 ; 001496.log size: 0 ; 001500.log size: 0 ; 001504.log size: 0 ; 001508.log size: 0 ; 001512.log size: 0 ; 001516.log size: 0 ; 001520.log size: 0 ; 001524.log size: 0 ; 001530.log size: 0 ; 001534.log size: 0 ; 001538.log size: 0 ; 001542.log size: 0 ; 001546.log size: 0 ; 001550.log size: 0 ; 001554.log size: 0 ; 001558.log size: 0 ; 001562.log size: 0 ; 001566.log size: 0 ; 001570.log size: 0 ; 001576.log size: 0 ; 001580.log size: 0 ; 001584.log size: 0 ; 001588.log size: 0 ; 001592.log size: 0 ; 001596.log size: 0 ; 001600.log size: 0 ; 001604.log size: 0 ; 001608.log size: 0 ; 001612.log size: 0 ; 001616.log size: 0 ; 001622.log size: 0 ; 001626.log size: 0 ; 001630.log size: 0 ; 001634.log size: 0 ; 001638.log size: 0 ; 001642.log size: 0 ; 001646.log size: 0 ; 001650.log size: 0 ; 001654.log size: 0 ; 001658.log size: 0 ; 001662.log size: 0 ; 001668.log size: 0 ; 001672.log size: 0 ; 001676.log size: 0 ; 001680.log size: 0 ; 001684.log size: 0 ; 001688.log size: 0 ; 001692.log size: 0 ; 001696.log size: 0 ; 001700.log size: 0 ; 001704.log size: 0 ; 001708.log size: 0 ; 001714.log size: 0 ; 001718.log size: 0 ; 001722.log size: 0 ; 001726.log size: 0 ; 001730.log size: 0 ; 001734.log size: 0 ; 001738.log size: 0 ; 001742.log size: 0 ; 001746.log size: 0 ; 001750.log size: 0 ; 001754.log size: 0 ; 001760.log size: 0 ; 001764.log size: 0 ; 001768.log size: 0 ; 001772.log size: 0 ; 001776.log size: 0 ; 001780.log size: 0 ; 001784.log size: 0 ; 001788.log size: 0 ; 001792.log size: 0 ; 001796.log size: 0 ; 001800.log size: 0 ; 001806.log size: 0 ; 001810.log size: 0 ; 001814.log size: 0 ; 001818.log size: 0 ; 001822.log size: 0 ; 001826.log size: 0 ; 001830.log size: 0 ; 001834.log size: 0 ; 001838.log size: 0 ; 001842.log size: 0 ; 001846.log size: 0 ; 001852.log size: 0 ; 001856.log size: 0 ; 001860.log size: 0 ; 001864.log size: 0 ; 001868.log size: 0 ; 001872.log size: 0 ; 001876.log size: 0 ; 001880.log size: 0 ; 001884.log size: 0 ; 001888.log size: 0 ; 001892.log size: 0 ; 001898.log size: 0 ; 001902.log size: 0 ; 001906.log size: 0 ; 001910.log size: 0 ; 001914.log size: 0 ; 001918.log size: 0 ; 001922.log size: 0 ; 001926.log size: 0 ; 001930.log size: 0 ; 001934.log size: 0 ; 001938.log size: 0 ; 001944.log size: 0 ; 001948.log size: 0 ; 001952.log size: 0 ; 001956.log size: 0 ; 001960.log size: 0 ; 001964.log size: 0 ; 001968.log size: 0 ; 001972.log size: 0 ; 001976.log size: 0 ; 001980.log size: 0 ; 001984.log size: 0 ; 001990.log size: 0 ; 001994.log size: 0 ; 001998.log size: 0 ; 002002.log size: 0 ; 002006.log size: 0 ; 002010.log size: 0 ; 002014.log size: 0 ; 002018.log size: 0 ; 002022.log size: 0 ; 002026.log size: 0 ; 002030.log size: 0 ; 002036.log size: 0 ; 002040.log size: 0 ; 002044.log size: 0 ; 002048.log size: 0 ; 002052.log size: 0 ; 002056.log size: 0 ; 002060.log size: 0 ; 002064.log size: 0 ; 002068.log size: 0 ; 002072.log size: 0 ; 002076.log size: 0 ; 002082.log size: 0 ; 002086.log size: 0 ; 002090.log size: 0 ; 002094.log size: 0 ; 002098.log size: 0 ; 002102.log size: 0 ; 002106.log size: 0 ; 002110.log size: 0 ; 002114.log size: 0 ; 002118.log size: 0 ; 002122.log size: 0 ; 002128.log size: 0 ; 002132.log size: 0 ; 002136.log size: 0 ; 002140.log size: 0 ; 002144.log size: 0 ; 002148.log size: 0 ; 002152.log size: 0 ; 002156.log size: 0 ; 002160.log size: 0 ; 002164.log size: 0 ; 002168.log size: 0 ; 002174.log size: 0 ; 002178.log size: 0 ; 002182.log size: 0 ; 002186.log size: 0 ; 002190.log size: 0 ; 002194.log size: 0 ; 002198.log size: 0 ; 002202.log size: 0 ; 002206.log size: 0 ; 002210.log size: 0 ;
2022/10/12-18:14:30.263254 4588350976 Options.error_if_exists: 0
2022/10/12-18:14:30.263267 4588350976 Options.create_if_missing: 0
2022/10/12-18:14:30.263272 4588350976 Options.paranoid_checks: 1
2022/10/12-18:14:30.263276 4588350976 Options.flush_verify_memtable_count: 1
2022/10/12-18:14:30.263280 4588350976 Options.track_and_verify_wals_in_manifest: 0
2022/10/12-18:14:30.263285 4588350976 Options.verify_sst_unique_id_in_manifest: 1
2022/10/12-18:14:30.263289 4588350976 Options.env: 0x7fd44ec51ef0
2022/10/12-18:14:30.263293 4588350976 Options.fs: LegacyFileSystem
2022/10/12-18:14:30.263298 4588350976 Options.info_log: 0x7fd450a04bc8
2022/10/12-18:14:30.263302 4588350976 Options.max_file_opening_threads: 16
2022/10/12-18:14:30.263306 4588350976 Options.statistics: 0x0
2022/10/12-18:14:30.263332 4588350976 Options.use_fsync: 0
2022/10/12-18:14:30.263339 4588350976 Options.max_log_file_size: 0
2022/10/12-18:14:30.263343 4588350976 Options.max_manifest_file_size: 1073741824
2022/10/12-18:14:30.263347 4588350976 Options.log_file_time_to_roll: 0
2022/10/12-18:14:30.263351 4588350976 Options.keep_log_file_num: 1000
2022/10/12-18:14:30.263356 4588350976 Options.recycle_log_file_num: 0
2022/10/12-18:14:30.263360 4588350976 Options.allow_fallocate: 1
2022/10/12-18:14:30.263382 4588350976 Options.allow_mmap_reads: 0
2022/10/12-18:14:30.263388 4588350976 Options.allow_mmap_writes: 0
2022/10/12-18:14:30.263392 4588350976 Options.use_direct_reads: 0
2022/10/12-18:14:30.263396 4588350976 Options.use_direct_io_for_flush_and_compaction: 0
2022/10/12-18:14:30.263400 4588350976 Options.create_missing_column_families: 0
2022/10/12-18:14:30.263404 4588350976 Options.db_log_dir:
2022/10/12-18:14:30.263408 4588350976 Options.wal_dir:
2022/10/12-18:14:30.263413 4588350976 Options.table_cache_numshardbits: 6
2022/10/12-18:14:30.263417 4588350976 Options.WAL_ttl_seconds: 0
2022/10/12-18:14:30.263421 4588350976 Options.WAL_size_limit_MB: 0
2022/10/12-18:14:30.263450 4588350976 Options.max_write_batch_group_size_bytes: 1048576
2022/10/12-18:14:30.263456 4588350976 Options.manifest_preallocation_size: 4194304
2022/10/12-18:14:30.263460 4588350976 Options.is_fd_close_on_exec: 1
2022/10/12-18:14:30.263464 4588350976 Options.advise_random_on_open: 1
2022/10/12-18:14:30.263468 4588350976 Options.db_write_buffer_size: 0
2022/10/12-18:14:30.263472 4588350976 Options.write_buffer_manager: 0x7fd450a06f40
2022/10/12-18:14:30.263476 4588350976 Options.access_hint_on_compaction_start: 1
2022/10/12-18:14:30.263480 4588350976 Options.random_access_max_buffer_size: 1048576
2022/10/12-18:14:30.263485 4588350976 Options.use_adaptive_mutex: 0
2022/10/12-18:14:30.263489 4588350976 Options.rate_limiter: 0x0
2022/10/12-18:14:30.263493 4588350976 Options.sst_file_manager.rate_bytes_per_sec: 0
2022/10/12-18:14:30.263498 4588350976 Options.wal_recovery_mode: 0
2022/10/12-18:14:30.263502 4588350976 Options.enable_thread_tracking: 0
2022/10/12-18:14:30.263506 4588350976 Options.enable_pipelined_write: 0
2022/10/12-18:14:30.263509 4588350976 Options.unordered_write: 0
2022/10/12-18:14:30.263513 4588350976 Options.allow_concurrent_memtable_write: 1
2022/10/12-18:14:30.263517 4588350976 Options.enable_write_thread_adaptive_yield: 1
2022/10/12-18:14:30.263521 4588350976 Options.write_thread_max_yield_usec: 100
2022/10/12-18:14:30.263525 4588350976 Options.write_thread_slow_yield_usec: 3
2022/10/12-18:14:30.263529 4588350976 Options.row_cache: None
2022/10/12-18:14:30.263534 4588350976 Options.wal_filter: None
2022/10/12-18:14:30.263538 4588350976 Options.avoid_flush_during_recovery: 0
2022/10/12-18:14:30.263542 4588350976 Options.allow_ingest_behind: 0
2022/10/12-18:14:30.263546 4588350976 Options.two_write_queues: 0
2022/10/12-18:14:30.263550 4588350976 Options.manual_wal_flush: 0
2022/10/12-18:14:30.263554 4588350976 Options.wal_compression: 0
2022/10/12-18:14:30.263558 4588350976 Options.atomic_flush: 0
2022/10/12-18:14:30.263562 4588350976 Options.avoid_unnecessary_blocking_io: 0
2022/10/12-18:14:30.263566 4588350976 Options.persist_stats_to_disk: 0
2022/10/12-18:14:30.263570 4588350976 Options.write_dbid_to_manifest: 0
2022/10/12-18:14:30.263574 4588350976 Options.log_readahead_size: 0
2022/10/12-18:14:30.263578 4588350976 Options.file_checksum_gen_factory: Unknown
2022/10/12-18:14:30.263582 4588350976 Options.best_efforts_recovery: 0
2022/10/12-18:14:30.263586 4588350976 Options.max_bgerror_resume_count: 2147483647
2022/10/12-18:14:30.263590 4588350976 Options.bgerror_resume_retry_interval: 1000000
2022/10/12-18:14:30.263620 4588350976 Options.allow_data_in_errors: 0
2022/10/12-18:14:30.263628 4588350976 Options.db_host_id: __hostname__
2022/10/12-18:14:30.263632 4588350976 Options.enforce_single_del_contracts: true
2022/10/12-18:14:30.263637 4588350976 Options.max_background_jobs: 2
2022/10/12-18:14:30.263641 4588350976 Options.max_background_compactions: -1
2022/10/12-18:14:30.263645 4588350976 Options.max_subcompactions: 1
2022/10/12-18:14:30.263649 4588350976 Options.avoid_flush_during_shutdown: 0
2022/10/12-18:14:30.263667 4588350976 Options.writable_file_max_buffer_size: 1048576
2022/10/12-18:14:30.263674 4588350976 Options.delayed_write_rate : 16777216
2022/10/12-18:14:30.263678 4588350976 Options.max_total_wal_size: 0
2022/10/12-18:14:30.263682 4588350976 Options.delete_obsolete_files_period_micros: 21600000000
2022/10/12-18:14:30.263687 4588350976 Options.stats_dump_period_sec: 600
2022/10/12-18:14:30.263691 4588350976 Options.stats_persist_period_sec: 600
2022/10/12-18:14:30.263695 4588350976 Options.stats_history_buffer_size: 1048576
2022/10/12-18:14:30.263699 4588350976 Options.max_open_files: -1
2022/10/12-18:14:30.263703 4588350976 Options.bytes_per_sync: 0
2022/10/12-18:14:30.263707 4588350976 Options.wal_bytes_per_sync: 0
2022/10/12-18:14:30.263711 4588350976 Options.strict_bytes_per_sync: 0
2022/10/12-18:14:30.263715 4588350976 Options.compaction_readahead_size: 0
2022/10/12-18:14:30.263719 4588350976 Options.max_background_flushes: -1
2022/10/12-18:14:30.263724 4588350976 Compression algorithms supported:
2022/10/12-18:14:30.263729 4588350976 kZSTD supported: 0
2022/10/12-18:14:30.263734 4588350976 kZlibCompression supported: 0
2022/10/12-18:14:30.263739 4588350976 kXpressCompression supported: 0
2022/10/12-18:14:30.263743 4588350976 kSnappyCompression supported: 0
2022/10/12-18:14:30.263748 4588350976 kZSTDNotFinalCompression supported: 0
2022/10/12-18:14:30.263752 4588350976 kLZ4HCCompression supported: 0
2022/10/12-18:14:30.263757 4588350976 kLZ4Compression supported: 0
2022/10/12-18:14:30.263761 4588350976 kBZip2Compression supported: 0
2022/10/12-18:14:30.263768 4588350976 Fast CRC32 supported: Supported on x86
2022/10/12-18:14:30.263773 4588350976 DMutex implementation: pthread_mutex_t
2022/10/12-18:14:30.266157 4588350976 [db/version_set.cc:5536] Recovering from manifest file: /tmp/rocksdbtest-502/db_test_8806_189362016023994854/MANIFEST-002211
2022/10/12-18:14:30.266397 4588350976 [db/column_family.cc:632] --------------- Options for column family [default]:
2022/10/12-18:14:30.266412 4588350976 Options.comparator: leveldb.BytewiseComparator
2022/10/12-18:14:30.266418 4588350976 Options.merge_operator: None
2022/10/12-18:14:30.266422 4588350976 Options.compaction_filter: None
2022/10/12-18:14:30.266427 4588350976 Options.compaction_filter_factory: None
2022/10/12-18:14:30.266432 4588350976 Options.sst_partitioner_factory: None
2022/10/12-18:14:30.266436 4588350976 Options.memtable_factory: SkipListFactory
2022/10/12-18:14:30.266441 4588350976 Options.table_factory: BlockBasedTable
2022/10/12-18:14:30.266469 4588350976 table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7fd450b044a0)
cache_index_and_filter_blocks: 0
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 1
index_type: 0
data_block_index_type: 0
index_shortening: 1
data_block_hash_table_util_ratio: 0.750000
checksum: 1
no_block_cache: 0
block_cache: 0x7fd450b040b8
block_cache_name: LRUCache
block_cache_options:
capacity : 8388608
num_shard_bits : 4
strict_capacity_limit : 0
memory_allocator : None
high_pri_pool_ratio: 0.000
low_pri_pool_ratio: 0.000
block_cache_compressed: 0x0
persistent_cache: 0x0
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: nullptr
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 5
enable_index_compression: 1
block_align: 0
max_auto_readahead_size: 262144
prepopulate_block_cache: 0
initial_auto_readahead_size: 8192
num_file_reads_for_auto_readahead: 2
2022/10/12-18:14:30.266479 4588350976 Options.write_buffer_size: 16752640
2022/10/12-18:14:30.266484 4588350976 Options.max_write_buffer_number: 2
2022/10/12-18:14:30.266490 4588350976 Options.compression: NoCompression
2022/10/12-18:14:30.266494 4588350976 Options.bottommost_compression: Disabled
2022/10/12-18:14:30.266499 4588350976 Options.prefix_extractor: nullptr
2022/10/12-18:14:30.266503 4588350976 Options.memtable_insert_with_hint_prefix_extractor: nullptr
2022/10/12-18:14:30.266507 4588350976 Options.num_levels: 7
2022/10/12-18:14:30.266511 4588350976 Options.min_write_buffer_number_to_merge: 1
2022/10/12-18:14:30.266526 4588350976 Options.max_write_buffer_number_to_maintain: 0
2022/10/12-18:14:30.266531 4588350976 Options.max_write_buffer_size_to_maintain: 0
2022/10/12-18:14:30.266535 4588350976 Options.bottommost_compression_opts.window_bits: -14
2022/10/12-18:14:30.266539 4588350976 Options.bottommost_compression_opts.level: 32767
2022/10/12-18:14:30.266543 4588350976 Options.bottommost_compression_opts.strategy: 0
2022/10/12-18:14:30.266547 4588350976 Options.bottommost_compression_opts.max_dict_bytes: 0
2022/10/12-18:14:30.266555 4588350976 Options.bottommost_compression_opts.zstd_max_train_bytes: 0
2022/10/12-18:14:30.266564 4588350976 Options.bottommost_compression_opts.parallel_threads: 1
2022/10/12-18:14:30.266572 4588350976 Options.bottommost_compression_opts.enabled: false
2022/10/12-18:14:30.266581 4588350976 Options.bottommost_compression_opts.max_dict_buffer_bytes: 0
2022/10/12-18:14:30.266590 4588350976 Options.bottommost_compression_opts.use_zstd_dict_trainer: true
2022/10/12-18:14:30.266599 4588350976 Options.compression_opts.window_bits: -14
2022/10/12-18:14:30.266606 4588350976 Options.compression_opts.level: 32767
2022/10/12-18:14:30.266610 4588350976 Options.compression_opts.strategy: 0
2022/10/12-18:14:30.266615 4588350976 Options.compression_opts.max_dict_bytes: 0
2022/10/12-18:14:30.266619 4588350976 Options.compression_opts.zstd_max_train_bytes: 0
2022/10/12-18:14:30.266623 4588350976 Options.compression_opts.use_zstd_dict_trainer: true
2022/10/12-18:14:30.266627 4588350976 Options.compression_opts.parallel_threads: 1
2022/10/12-18:14:30.266632 4588350976 Options.compression_opts.enabled: false
2022/10/12-18:14:30.266636 4588350976 Options.compression_opts.max_dict_buffer_bytes: 0
2022/10/12-18:14:30.266640 4588350976 Options.level0_file_num_compaction_trigger: 6
2022/10/12-18:14:30.266644 4588350976 Options.level0_slowdown_writes_trigger: 2147483647
2022/10/12-18:14:30.266648 4588350976 Options.level0_stop_writes_trigger: 2147483647
2022/10/12-18:14:30.266652 4588350976 Options.target_file_size_base: 2097152
2022/10/12-18:14:30.266656 4588350976 Options.target_file_size_multiplier: 1
2022/10/12-18:14:30.266661 4588350976 Options.max_bytes_for_level_base: 10485760
2022/10/12-18:14:30.266665 4588350976 Options.level_compaction_dynamic_level_bytes: 0
2022/10/12-18:14:30.266669 4588350976 Options.max_bytes_for_level_multiplier: 10.000000
2022/10/12-18:14:30.266674 4588350976 Options.max_bytes_for_level_multiplier_addtl[0]: 1
2022/10/12-18:14:30.266679 4588350976 Options.max_bytes_for_level_multiplier_addtl[1]: 1
2022/10/12-18:14:30.266683 4588350976 Options.max_bytes_for_level_multiplier_addtl[2]: 1
2022/10/12-18:14:30.266688 4588350976 Options.max_bytes_for_level_multiplier_addtl[3]: 1
2022/10/12-18:14:30.266697 4588350976 Options.max_bytes_for_level_multiplier_addtl[4]: 1
2022/10/12-18:14:30.266706 4588350976 Options.max_bytes_for_level_multiplier_addtl[5]: 1
2022/10/12-18:14:30.266715 4588350976 Options.max_bytes_for_level_multiplier_addtl[6]: 1
2022/10/12-18:14:30.266726 4588350976 Options.max_sequential_skip_in_iterations: 8
2022/10/12-18:14:30.266735 4588350976 Options.max_compaction_bytes: 52428800
2022/10/12-18:14:30.266744 4588350976 Options.arena_block_size: 1048576
2022/10/12-18:14:30.266752 4588350976 Options.soft_pending_compaction_bytes_limit: 68719476736
2022/10/12-18:14:30.266759 4588350976 Options.hard_pending_compaction_bytes_limit: 274877906944
2022/10/12-18:14:30.266764 4588350976 Options.disable_auto_compactions: 0
2022/10/12-18:14:30.266770 4588350976 Options.compaction_style: kCompactionStyleFIFO
2022/10/12-18:14:30.266776 4588350976 Options.compaction_pri: kByCompensatedSize
2022/10/12-18:14:30.266780 4588350976 Options.compaction_options_universal.size_ratio: 1
2022/10/12-18:14:30.266784 4588350976 Options.compaction_options_universal.min_merge_width: 2
2022/10/12-18:14:30.266788 4588350976 Options.compaction_options_universal.max_merge_width: 4294967295
2022/10/12-18:14:30.266793 4588350976 Options.compaction_options_universal.max_size_amplification_percent: 200
2022/10/12-18:14:30.266797 4588350976 Options.compaction_options_universal.compression_size_percent: -1
2022/10/12-18:14:30.266803 4588350976 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
2022/10/12-18:14:30.266807 4588350976 Options.compaction_options_fifo.max_table_files_size: 5169
2022/10/12-18:14:30.266811 4588350976 Options.compaction_options_fifo.allow_compaction: 0
2022/10/12-18:14:30.266818 4588350976 Options.table_properties_collectors:
2022/10/12-18:14:30.266823 4588350976 Options.inplace_update_support: 0
2022/10/12-18:14:30.266827 4588350976 Options.inplace_update_num_locks: 10000
2022/10/12-18:14:30.266831 4588350976 Options.memtable_prefix_bloom_size_ratio: 0.000000
2022/10/12-18:14:30.266836 4588350976 Options.memtable_whole_key_filtering: 0
2022/10/12-18:14:30.266840 4588350976 Options.memtable_huge_page_size: 0
2022/10/12-18:14:30.266844 4588350976 Options.bloom_locality: 0
2022/10/12-18:14:30.266849 4588350976 Options.max_successive_merges: 0
2022/10/12-18:14:30.266853 4588350976 Options.optimize_filters_for_hits: 0
2022/10/12-18:14:30.266857 4588350976 Options.paranoid_file_checks: 0
2022/10/12-18:14:30.266861 4588350976 Options.force_consistency_checks: 1
2022/10/12-18:14:30.266865 4588350976 Options.report_bg_io_stats: 0
2022/10/12-18:14:30.266870 4588350976 Options.ttl: 2592000
2022/10/12-18:14:30.266874 4588350976 Options.periodic_compaction_seconds: 2592000
2022/10/12-18:14:30.266878 4588350976 Options.preclude_last_level_data_seconds: 0
2022/10/12-18:14:30.266882 4588350976 Options.preserve_internal_time_seconds: 0
2022/10/12-18:14:30.266886 4588350976 Options.enable_blob_files: false
2022/10/12-18:14:30.266890 4588350976 Options.min_blob_size: 0
2022/10/12-18:14:30.266894 4588350976 Options.blob_file_size: 268435456
2022/10/12-18:14:30.266899 4588350976 Options.blob_compression_type: NoCompression
2022/10/12-18:14:30.266903 4588350976 Options.enable_blob_garbage_collection: false
2022/10/12-18:14:30.266907 4588350976 Options.blob_garbage_collection_age_cutoff: 0.250000
2022/10/12-18:14:30.266912 4588350976 Options.blob_garbage_collection_force_threshold: 1.000000
2022/10/12-18:14:30.266917 4588350976 Options.blob_compaction_readahead_size: 0
2022/10/12-18:14:30.266921 4588350976 Options.blob_file_starting_level: 0
2022/10/12-18:14:30.266926 4588350976 Options.experimental_mempurge_threshold: 0.000000
2022/10/12-18:14:30.267201 4588350976 [db/column_family.cc:632] --------------- Options for column family [pikachu]:
2022/10/12-18:14:30.267214 4588350976 Options.comparator: leveldb.BytewiseComparator
2022/10/12-18:14:30.267219 4588350976 Options.merge_operator: None
2022/10/12-18:14:30.267224 4588350976 Options.compaction_filter: None
2022/10/12-18:14:30.267228 4588350976 Options.compaction_filter_factory: None
2022/10/12-18:14:30.267232 4588350976 Options.sst_partitioner_factory: None
2022/10/12-18:14:30.267237 4588350976 Options.memtable_factory: SkipListFactory
2022/10/12-18:14:30.267241 4588350976 Options.table_factory: BlockBasedTable
2022/10/12-18:14:30.267266 4588350976 table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7fd450b044a0)
cache_index_and_filter_blocks: 0
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 1
index_type: 0
data_block_index_type: 0
index_shortening: 1
data_block_hash_table_util_ratio: 0.750000
checksum: 1
no_block_cache: 0
block_cache: 0x7fd450b040b8
block_cache_name: LRUCache
block_cache_options:
capacity : 8388608
num_shard_bits : 4
strict_capacity_limit : 0
memory_allocator : None
high_pri_pool_ratio: 0.000
low_pri_pool_ratio: 0.000
block_cache_compressed: 0x0
persistent_cache: 0x0
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: nullptr
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 5
enable_index_compression: 1
block_align: 0
max_auto_readahead_size: 262144
prepopulate_block_cache: 0
initial_auto_readahead_size: 8192
num_file_reads_for_auto_readahead: 2
2022/10/12-18:14:30.267276 4588350976 Options.write_buffer_size: 16752640
2022/10/12-18:14:30.267281 4588350976 Options.max_write_buffer_number: 2
2022/10/12-18:14:30.267286 4588350976 Options.compression: NoCompression
2022/10/12-18:14:30.267291 4588350976 Options.bottommost_compression: Disabled
2022/10/12-18:14:30.267295 4588350976 Options.prefix_extractor: nullptr
2022/10/12-18:14:30.267299 4588350976 Options.memtable_insert_with_hint_prefix_extractor: nullptr
2022/10/12-18:14:30.267303 4588350976 Options.num_levels: 7
2022/10/12-18:14:30.267307 4588350976 Options.min_write_buffer_number_to_merge: 1
2022/10/12-18:14:30.267311 4588350976 Options.max_write_buffer_number_to_maintain: 0
2022/10/12-18:14:30.267315 4588350976 Options.max_write_buffer_size_to_maintain: 0
2022/10/12-18:14:30.267319 4588350976 Options.bottommost_compression_opts.window_bits: -14
2022/10/12-18:14:30.267324 4588350976 Options.bottommost_compression_opts.level: 32767
2022/10/12-18:14:30.267328 4588350976 Options.bottommost_compression_opts.strategy: 0
2022/10/12-18:14:30.267332 4588350976 Options.bottommost_compression_opts.max_dict_bytes: 0
2022/10/12-18:14:30.267336 4588350976 Options.bottommost_compression_opts.zstd_max_train_bytes: 0
2022/10/12-18:14:30.267340 4588350976 Options.bottommost_compression_opts.parallel_threads: 1
2022/10/12-18:14:30.267344 4588350976 Options.bottommost_compression_opts.enabled: false
2022/10/12-18:14:30.267348 4588350976 Options.bottommost_compression_opts.max_dict_buffer_bytes: 0
2022/10/12-18:14:30.267352 4588350976 Options.bottommost_compression_opts.use_zstd_dict_trainer: true
2022/10/12-18:14:30.267356 4588350976 Options.compression_opts.window_bits: -14
2022/10/12-18:14:30.267361 4588350976 Options.compression_opts.level: 32767
2022/10/12-18:14:30.267365 4588350976 Options.compression_opts.strategy: 0
2022/10/12-18:14:30.267369 4588350976 Options.compression_opts.max_dict_bytes: 0
2022/10/12-18:14:30.267373 4588350976 Options.compression_opts.zstd_max_train_bytes: 0
2022/10/12-18:14:30.267377 4588350976 Options.compression_opts.use_zstd_dict_trainer: true
2022/10/12-18:14:30.267381 4588350976 Options.compression_opts.parallel_threads: 1
2022/10/12-18:14:30.267385 4588350976 Options.compression_opts.enabled: false
2022/10/12-18:14:30.267389 4588350976 Options.compression_opts.max_dict_buffer_bytes: 0
2022/10/12-18:14:30.267393 4588350976 Options.level0_file_num_compaction_trigger: 6
2022/10/12-18:14:30.267397 4588350976 Options.level0_slowdown_writes_trigger: 2147483647
2022/10/12-18:14:30.267401 4588350976 Options.level0_stop_writes_trigger: 2147483647
2022/10/12-18:14:30.267405 4588350976 Options.target_file_size_base: 2097152
2022/10/12-18:14:30.267484 4588350976 Options.target_file_size_multiplier: 1
2022/10/12-18:14:30.267494 4588350976 Options.max_bytes_for_level_base: 10485760
2022/10/12-18:14:30.267501 4588350976 Options.level_compaction_dynamic_level_bytes: 0
2022/10/12-18:14:30.267505 4588350976 Options.max_bytes_for_level_multiplier: 10.000000
2022/10/12-18:14:30.267511 4588350976 Options.max_bytes_for_level_multiplier_addtl[0]: 1
2022/10/12-18:14:30.267515 4588350976 Options.max_bytes_for_level_multiplier_addtl[1]: 1
2022/10/12-18:14:30.267520 4588350976 Options.max_bytes_for_level_multiplier_addtl[2]: 1
2022/10/12-18:14:30.267524 4588350976 Options.max_bytes_for_level_multiplier_addtl[3]: 1
2022/10/12-18:14:30.267528 4588350976 Options.max_bytes_for_level_multiplier_addtl[4]: 1
2022/10/12-18:14:30.267532 4588350976 Options.max_bytes_for_level_multiplier_addtl[5]: 1
2022/10/12-18:14:30.267536 4588350976 Options.max_bytes_for_level_multiplier_addtl[6]: 1
2022/10/12-18:14:30.267541 4588350976 Options.max_sequential_skip_in_iterations: 8
2022/10/12-18:14:30.267545 4588350976 Options.max_compaction_bytes: 52428800
2022/10/12-18:14:30.267549 4588350976 Options.arena_block_size: 1048576
2022/10/12-18:14:30.267553 4588350976 Options.soft_pending_compaction_bytes_limit: 68719476736
2022/10/12-18:14:30.267557 4588350976 Options.hard_pending_compaction_bytes_limit: 274877906944
2022/10/12-18:14:30.267561 4588350976 Options.disable_auto_compactions: 0
2022/10/12-18:14:30.267567 4588350976 Options.compaction_style: kCompactionStyleFIFO
2022/10/12-18:14:30.267573 4588350976 Options.compaction_pri: kByCompensatedSize
2022/10/12-18:14:30.267577 4588350976 Options.compaction_options_universal.size_ratio: 1
2022/10/12-18:14:30.267581 4588350976 Options.compaction_options_universal.min_merge_width: 2
2022/10/12-18:14:30.267585 4588350976 Options.compaction_options_universal.max_merge_width: 4294967295
2022/10/12-18:14:30.267589 4588350976 Options.compaction_options_universal.max_size_amplification_percent: 200
2022/10/12-18:14:30.267593 4588350976 Options.compaction_options_universal.compression_size_percent: -1
2022/10/12-18:14:30.267599 4588350976 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
2022/10/12-18:14:30.267603 4588350976 Options.compaction_options_fifo.max_table_files_size: 5169
2022/10/12-18:14:30.267608 4588350976 Options.compaction_options_fifo.allow_compaction: 0
2022/10/12-18:14:30.267614 4588350976 Options.table_properties_collectors:
2022/10/12-18:14:30.267619 4588350976 Options.inplace_update_support: 0
2022/10/12-18:14:30.267623 4588350976 Options.inplace_update_num_locks: 10000
2022/10/12-18:14:30.267627 4588350976 Options.memtable_prefix_bloom_size_ratio: 0.000000
2022/10/12-18:14:30.267632 4588350976 Options.memtable_whole_key_filtering: 0
2022/10/12-18:14:30.267636 4588350976 Options.memtable_huge_page_size: 0
2022/10/12-18:14:30.267640 4588350976 Options.bloom_locality: 0
2022/10/12-18:14:30.267644 4588350976 Options.max_successive_merges: 0
2022/10/12-18:14:30.267648 4588350976 Options.optimize_filters_for_hits: 0
2022/10/12-18:14:30.267652 4588350976 Options.paranoid_file_checks: 0
2022/10/12-18:14:30.267656 4588350976 Options.force_consistency_checks: 1
2022/10/12-18:14:30.267660 4588350976 Options.report_bg_io_stats: 0
2022/10/12-18:14:30.267664 4588350976 Options.ttl: 2592000
2022/10/12-18:14:30.267668 4588350976 Options.periodic_compaction_seconds: 2592000
2022/10/12-18:14:30.267673 4588350976 Options.preclude_last_level_data_seconds: 0
2022/10/12-18:14:30.267677 4588350976 Options.preserve_internal_time_seconds: 0
2022/10/12-18:14:30.267681 4588350976 Options.enable_blob_files: false
2022/10/12-18:14:30.267685 4588350976 Options.min_blob_size: 0
2022/10/12-18:14:30.267689 4588350976 Options.blob_file_size: 268435456
2022/10/12-18:14:30.267694 4588350976 Options.blob_compression_type: NoCompression
2022/10/12-18:14:30.267700 4588350976 Options.enable_blob_garbage_collection: false
2022/10/12-18:14:30.267708 4588350976 Options.blob_garbage_collection_age_cutoff: 0.250000
2022/10/12-18:14:30.267717 4588350976 Options.blob_garbage_collection_force_threshold: 1.000000
2022/10/12-18:14:30.267722 4588350976 Options.blob_compaction_readahead_size: 0
2022/10/12-18:14:30.267726 4588350976 Options.blob_file_starting_level: 0
2022/10/12-18:14:30.267730 4588350976 Options.experimental_mempurge_threshold: 0.000000
2022/10/12-18:14:30.274811 4588350976 [db/version_set.cc:5584] Recovered from manifest file:/tmp/rocksdbtest-502/db_test_8806_189362016023994854/MANIFEST-002211 succeeded,manifest_file_number is 2211, next_file_number is 2213, last_sequence is 20, log_number is 2207,prev_log_number is 0,max_column_family is 1,min_log_number_to_keep is 1480
2022/10/12-18:14:30.274830 4588350976 [db/version_set.cc:5593] Column family [default] (ID 0), log number is 2207
2022/10/12-18:14:30.274836 4588350976 [db/version_set.cc:5593] Column family [pikachu] (ID 1), log number is 2207
2022/10/12-18:14:30.274967 4588350976 [db/db_impl/db_impl_open.cc:538] DB ID: c7a33b65-632c-48ac-ae65-5a76ba45e7e1
2022/10/12-18:14:30.295547 4588350976 EVENT_LOG_v1 {"time_micros": 1665598470295452, "job": 1, "event": "recovery_started", "wal_files": [1480, 1484, 1488, 1492, 1496, 1500, 1504, 1508, 1512, 1516, 1520, 1524, 1530, 1534, 1538, 1542, 1546, 1550, 1554, 1558, 1562, 1566, 1570, 1576, 1580, 1584, 1588, 1592, 1596, 1600, 1604, 1608, 1612, 1616, 1622, 1626, 1630, 1634, 1638, 1642, 1646, 1650, 1654, 1658, 1662, 1668, 1672, 1676, 1680, 1684, 1688, 1692, 1696, 1700, 1704, 1708, 1714, 1718, 1722, 1726, 1730, 1734, 1738, 1742, 1746, 1750, 1754, 1760, 1764, 1768, 1772, 1776, 1780, 1784, 1788, 1792, 1796, 1800, 1806, 1810, 1814, 1818, 1822, 1826, 1830, 1834, 1838, 1842, 1846, 1852, 1856, 1860, 1864, 1868, 1872, 1876, 1880, 1884, 1888, 1892, 1898, 1902, 1906, 1910, 1914, 1918, 1922, 1926, 1930, 1934, 1938, 1944, 1948, 1952, 1956, 1960, 1964, 1968, 1972, 1976, 1980, 1984, 1990, 1994, 1998, 2002, 2006, 2010, 2014, 2018, 2022, 2026, 2030, 2036, 2040, 2044, 2048, 2052, 2056, 2060, 2064, 2068, 2072, 2076, 2082, 2086, 2090, 2094, 2098, 2102, 2106, 2110, 2114, 2118, 2122, 2128, 2132, 2136, 2140, 2144, 2148, 2152, 2156, 2160, 2164, 2168, 2174, 2178, 2182, 2186, 2190, 2194, 2198, 2202, 2206, 2210]}
2022/10/12-18:14:30.295567 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1480 since it is older than min log to keep #2207
2022/10/12-18:14:30.295574 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1484 since it is older than min log to keep #2207
2022/10/12-18:14:30.295579 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1488 since it is older than min log to keep #2207
2022/10/12-18:14:30.295583 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1492 since it is older than min log to keep #2207
2022/10/12-18:14:30.295588 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1496 since it is older than min log to keep #2207
2022/10/12-18:14:30.295592 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1500 since it is older than min log to keep #2207
2022/10/12-18:14:30.295597 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1504 since it is older than min log to keep #2207
2022/10/12-18:14:30.295601 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1508 since it is older than min log to keep #2207
2022/10/12-18:14:30.295606 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1512 since it is older than min log to keep #2207
2022/10/12-18:14:30.295610 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1516 since it is older than min log to keep #2207
2022/10/12-18:14:30.295614 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1520 since it is older than min log to keep #2207
2022/10/12-18:14:30.295619 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1524 since it is older than min log to keep #2207
2022/10/12-18:14:30.295623 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1530 since it is older than min log to keep #2207
2022/10/12-18:14:30.295627 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1534 since it is older than min log to keep #2207
2022/10/12-18:14:30.295632 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1538 since it is older than min log to keep #2207
2022/10/12-18:14:30.295636 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1542 since it is older than min log to keep #2207
2022/10/12-18:14:30.295641 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1546 since it is older than min log to keep #2207
2022/10/12-18:14:30.295645 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1550 since it is older than min log to keep #2207
2022/10/12-18:14:30.295649 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1554 since it is older than min log to keep #2207
2022/10/12-18:14:30.295654 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1558 since it is older than min log to keep #2207
2022/10/12-18:14:30.295658 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1562 since it is older than min log to keep #2207
2022/10/12-18:14:30.295662 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1566 since it is older than min log to keep #2207
2022/10/12-18:14:30.295667 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1570 since it is older than min log to keep #2207
2022/10/12-18:14:30.295671 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1576 since it is older than min log to keep #2207
2022/10/12-18:14:30.295676 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1580 since it is older than min log to keep #2207
2022/10/12-18:14:30.295680 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1584 since it is older than min log to keep #2207
2022/10/12-18:14:30.295684 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1588 since it is older than min log to keep #2207
2022/10/12-18:14:30.295689 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1592 since it is older than min log to keep #2207
2022/10/12-18:14:30.295693 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1596 since it is older than min log to keep #2207
2022/10/12-18:14:30.295697 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1600 since it is older than min log to keep #2207
2022/10/12-18:14:30.295702 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1604 since it is older than min log to keep #2207
2022/10/12-18:14:30.295706 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1608 since it is older than min log to keep #2207
2022/10/12-18:14:30.295710 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1612 since it is older than min log to keep #2207
2022/10/12-18:14:30.295715 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1616 since it is older than min log to keep #2207
2022/10/12-18:14:30.295719 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1622 since it is older than min log to keep #2207
2022/10/12-18:14:30.295723 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1626 since it is older than min log to keep #2207
2022/10/12-18:14:30.295728 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1630 since it is older than min log to keep #2207
2022/10/12-18:14:30.295732 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1634 since it is older than min log to keep #2207
2022/10/12-18:14:30.295736 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1638 since it is older than min log to keep #2207
2022/10/12-18:14:30.295740 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1642 since it is older than min log to keep #2207
2022/10/12-18:14:30.295745 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1646 since it is older than min log to keep #2207
2022/10/12-18:14:30.295749 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1650 since it is older than min log to keep #2207
2022/10/12-18:14:30.295753 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1654 since it is older than min log to keep #2207
2022/10/12-18:14:30.295758 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1658 since it is older than min log to keep #2207
2022/10/12-18:14:30.295762 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1662 since it is older than min log to keep #2207
2022/10/12-18:14:30.295766 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1668 since it is older than min log to keep #2207
2022/10/12-18:14:30.295771 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1672 since it is older than min log to keep #2207
2022/10/12-18:14:30.295775 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1676 since it is older than min log to keep #2207
2022/10/12-18:14:30.295779 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1680 since it is older than min log to keep #2207
2022/10/12-18:14:30.295784 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1684 since it is older than min log to keep #2207
2022/10/12-18:14:30.295788 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1688 since it is older than min log to keep #2207
2022/10/12-18:14:30.295792 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1692 since it is older than min log to keep #2207
2022/10/12-18:14:30.295797 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1696 since it is older than min log to keep #2207
2022/10/12-18:14:30.295801 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1700 since it is older than min log to keep #2207
2022/10/12-18:14:30.295805 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1704 since it is older than min log to keep #2207
2022/10/12-18:14:30.295810 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1708 since it is older than min log to keep #2207
2022/10/12-18:14:30.295814 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1714 since it is older than min log to keep #2207
2022/10/12-18:14:30.295818 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1718 since it is older than min log to keep #2207
2022/10/12-18:14:30.295823 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1722 since it is older than min log to keep #2207
2022/10/12-18:14:30.295827 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1726 since it is older than min log to keep #2207
2022/10/12-18:14:30.295831 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1730 since it is older than min log to keep #2207
2022/10/12-18:14:30.295836 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1734 since it is older than min log to keep #2207
2022/10/12-18:14:30.295840 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1738 since it is older than min log to keep #2207
2022/10/12-18:14:30.295844 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1742 since it is older than min log to keep #2207
2022/10/12-18:14:30.295848 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1746 since it is older than min log to keep #2207
2022/10/12-18:14:30.295853 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1750 since it is older than min log to keep #2207
2022/10/12-18:14:30.295857 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1754 since it is older than min log to keep #2207
2022/10/12-18:14:30.295861 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1760 since it is older than min log to keep #2207
2022/10/12-18:14:30.295866 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1764 since it is older than min log to keep #2207
2022/10/12-18:14:30.295870 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1768 since it is older than min log to keep #2207
2022/10/12-18:14:30.295874 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1772 since it is older than min log to keep #2207
2022/10/12-18:14:30.295879 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1776 since it is older than min log to keep #2207
2022/10/12-18:14:30.295883 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1780 since it is older than min log to keep #2207
2022/10/12-18:14:30.295887 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1784 since it is older than min log to keep #2207
2022/10/12-18:14:30.295892 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1788 since it is older than min log to keep #2207
2022/10/12-18:14:30.295896 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1792 since it is older than min log to keep #2207
2022/10/12-18:14:30.295900 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1796 since it is older than min log to keep #2207
2022/10/12-18:14:30.295905 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1800 since it is older than min log to keep #2207
2022/10/12-18:14:30.295909 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1806 since it is older than min log to keep #2207
2022/10/12-18:14:30.295913 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1810 since it is older than min log to keep #2207
2022/10/12-18:14:30.295918 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1814 since it is older than min log to keep #2207
2022/10/12-18:14:30.295922 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1818 since it is older than min log to keep #2207
2022/10/12-18:14:30.295926 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1822 since it is older than min log to keep #2207
2022/10/12-18:14:30.295931 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1826 since it is older than min log to keep #2207
2022/10/12-18:14:30.295935 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1830 since it is older than min log to keep #2207
2022/10/12-18:14:30.295939 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1834 since it is older than min log to keep #2207
2022/10/12-18:14:30.295944 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1838 since it is older than min log to keep #2207
2022/10/12-18:14:30.295948 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1842 since it is older than min log to keep #2207
2022/10/12-18:14:30.295952 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1846 since it is older than min log to keep #2207
2022/10/12-18:14:30.295957 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1852 since it is older than min log to keep #2207
2022/10/12-18:14:30.295961 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1856 since it is older than min log to keep #2207
2022/10/12-18:14:30.295965 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1860 since it is older than min log to keep #2207
2022/10/12-18:14:30.295970 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1864 since it is older than min log to keep #2207
2022/10/12-18:14:30.295974 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1868 since it is older than min log to keep #2207
2022/10/12-18:14:30.295978 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1872 since it is older than min log to keep #2207
2022/10/12-18:14:30.295983 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1876 since it is older than min log to keep #2207
2022/10/12-18:14:30.295987 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1880 since it is older than min log to keep #2207
2022/10/12-18:14:30.295991 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1884 since it is older than min log to keep #2207
2022/10/12-18:14:30.295995 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1888 since it is older than min log to keep #2207
2022/10/12-18:14:30.296000 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1892 since it is older than min log to keep #2207
2022/10/12-18:14:30.296004 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1898 since it is older than min log to keep #2207
2022/10/12-18:14:30.296008 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1902 since it is older than min log to keep #2207
2022/10/12-18:14:30.296012 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1906 since it is older than min log to keep #2207
2022/10/12-18:14:30.296017 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1910 since it is older than min log to keep #2207
2022/10/12-18:14:30.296021 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1914 since it is older than min log to keep #2207
2022/10/12-18:14:30.296025 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1918 since it is older than min log to keep #2207
2022/10/12-18:14:30.296029 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1922 since it is older than min log to keep #2207
2022/10/12-18:14:30.296034 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1926 since it is older than min log to keep #2207
2022/10/12-18:14:30.296038 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1930 since it is older than min log to keep #2207
2022/10/12-18:14:30.296043 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1934 since it is older than min log to keep #2207
2022/10/12-18:14:30.296047 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1938 since it is older than min log to keep #2207
2022/10/12-18:14:30.296051 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1944 since it is older than min log to keep #2207
2022/10/12-18:14:30.296056 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1948 since it is older than min log to keep #2207
2022/10/12-18:14:30.296060 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1952 since it is older than min log to keep #2207
2022/10/12-18:14:30.296064 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1956 since it is older than min log to keep #2207
2022/10/12-18:14:30.296068 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1960 since it is older than min log to keep #2207
2022/10/12-18:14:30.296073 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1964 since it is older than min log to keep #2207
2022/10/12-18:14:30.296077 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1968 since it is older than min log to keep #2207
2022/10/12-18:14:30.296081 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1972 since it is older than min log to keep #2207
2022/10/12-18:14:30.296086 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1976 since it is older than min log to keep #2207
2022/10/12-18:14:30.296090 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1980 since it is older than min log to keep #2207
2022/10/12-18:14:30.296094 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1984 since it is older than min log to keep #2207
2022/10/12-18:14:30.296099 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1990 since it is older than min log to keep #2207
2022/10/12-18:14:30.296103 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1994 since it is older than min log to keep #2207
2022/10/12-18:14:30.296107 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #1998 since it is older than min log to keep #2207
2022/10/12-18:14:30.296112 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2002 since it is older than min log to keep #2207
2022/10/12-18:14:30.296116 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2006 since it is older than min log to keep #2207
2022/10/12-18:14:30.296120 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2010 since it is older than min log to keep #2207
2022/10/12-18:14:30.296125 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2014 since it is older than min log to keep #2207
2022/10/12-18:14:30.296198 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2018 since it is older than min log to keep #2207
2022/10/12-18:14:30.296269 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2022 since it is older than min log to keep #2207
2022/10/12-18:14:30.296275 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2026 since it is older than min log to keep #2207
2022/10/12-18:14:30.296280 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2030 since it is older than min log to keep #2207
2022/10/12-18:14:30.296285 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2036 since it is older than min log to keep #2207
2022/10/12-18:14:30.296290 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2040 since it is older than min log to keep #2207
2022/10/12-18:14:30.296329 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2044 since it is older than min log to keep #2207
2022/10/12-18:14:30.296387 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2048 since it is older than min log to keep #2207
2022/10/12-18:14:30.296394 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2052 since it is older than min log to keep #2207
2022/10/12-18:14:30.296399 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2056 since it is older than min log to keep #2207
2022/10/12-18:14:30.296404 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2060 since it is older than min log to keep #2207
2022/10/12-18:14:30.296408 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2064 since it is older than min log to keep #2207
2022/10/12-18:14:30.296413 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2068 since it is older than min log to keep #2207
2022/10/12-18:14:30.296418 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2072 since it is older than min log to keep #2207
2022/10/12-18:14:30.296422 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2076 since it is older than min log to keep #2207
2022/10/12-18:14:30.296429 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2082 since it is older than min log to keep #2207
2022/10/12-18:14:30.296437 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2086 since it is older than min log to keep #2207
2022/10/12-18:14:30.296442 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2090 since it is older than min log to keep #2207
2022/10/12-18:14:30.296447 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2094 since it is older than min log to keep #2207
2022/10/12-18:14:30.296451 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2098 since it is older than min log to keep #2207
2022/10/12-18:14:30.296456 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2102 since it is older than min log to keep #2207
2022/10/12-18:14:30.296461 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2106 since it is older than min log to keep #2207
2022/10/12-18:14:30.296465 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2110 since it is older than min log to keep #2207
2022/10/12-18:14:30.296470 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2114 since it is older than min log to keep #2207
2022/10/12-18:14:30.296475 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2118 since it is older than min log to keep #2207
2022/10/12-18:14:30.296479 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2122 since it is older than min log to keep #2207
2022/10/12-18:14:30.296522 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2128 since it is older than min log to keep #2207
2022/10/12-18:14:30.296552 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2132 since it is older than min log to keep #2207
2022/10/12-18:14:30.299644 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2136 since it is older than min log to keep #2207
2022/10/12-18:14:30.299654 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2140 since it is older than min log to keep #2207
2022/10/12-18:14:30.299660 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2144 since it is older than min log to keep #2207
2022/10/12-18:14:30.299664 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2148 since it is older than min log to keep #2207
2022/10/12-18:14:30.299669 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2152 since it is older than min log to keep #2207
2022/10/12-18:14:30.299674 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2156 since it is older than min log to keep #2207
2022/10/12-18:14:30.299678 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2160 since it is older than min log to keep #2207
2022/10/12-18:14:30.299683 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2164 since it is older than min log to keep #2207
2022/10/12-18:14:30.299687 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2168 since it is older than min log to keep #2207
2022/10/12-18:14:30.299692 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2174 since it is older than min log to keep #2207
2022/10/12-18:14:30.299696 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2178 since it is older than min log to keep #2207
2022/10/12-18:14:30.299701 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2182 since it is older than min log to keep #2207
2022/10/12-18:14:30.299705 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2186 since it is older than min log to keep #2207
2022/10/12-18:14:30.299710 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2190 since it is older than min log to keep #2207
2022/10/12-18:14:30.299714 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2194 since it is older than min log to keep #2207
2022/10/12-18:14:30.299719 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2198 since it is older than min log to keep #2207
2022/10/12-18:14:30.299723 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2202 since it is older than min log to keep #2207
2022/10/12-18:14:30.299728 4588350976 [db/db_impl/db_impl_open.cc:1031] Skipping log #2206 since it is older than min log to keep #2207
2022/10/12-18:14:30.299736 4588350976 [db/db_impl/db_impl_open.cc:1044] Recovering log #2210 mode 0
2022/10/12-18:14:30.304645 4588350976 EVENT_LOG_v1 {"time_micros": 1665598470304634, "job": 1, "event": "recovery_finished"}
2022/10/12-18:14:30.341998 4588350976 [db/version_set.cc:5056] Creating manifest 2215
2022/10/12-18:14:30.405803 4588350976 [DEBUG] [db/db_impl/db_impl_files.cc:367] [JOB 2] Delete /tmp/rocksdbtest-502/db_test_8806_189362016023994854/MANIFEST-002211 type=3 #2211 -- OK
2022/10/12-18:14:30.413186 4588350976 [db/db_impl/db_impl_open.cc:1988] SstFileManager instance 0x7fd44ed05730
2022/10/12-18:14:30.415549 4588350976 DB pointer 0x7fd451808800
2022/10/12-18:14:30.417815 123145507098624 (Original Log Time 2022/10/12-18:14:30.413344) [db/compaction/compaction_picker_fifo.cc:226] [pikachu] FIFO compaction: picking file 1305 with size 1034B for deletion
2022/10/12-18:14:30.417830 123145507098624 (Original Log Time 2022/10/12-18:14:30.417717) [db/db_impl/db_impl_compaction_flush.cc:3302] [pikachu] Deleted 1 files
2022/10/12-18:14:30.419265 4588350976 [db/db_impl/db_impl_compaction_flush.cc:1895] [pikachu] Manual compaction waiting for all other scheduled background compactions to finish
2022/10/12-18:14:30.423014 123145507098624 [file/delete_scheduler.cc:77] Deleted file /tmp/rocksdbtest-502/db_test_8806_189362016023994854/001305.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
2022/10/12-18:14:30.423033 123145507098624 [DEBUG] [db/db_impl/db_impl_files.cc:367] [JOB 3] Delete /tmp/rocksdbtest-502/db_test_8806_189362016023994854/001305.sst type=2 #1305 -- OK
2022/10/12-18:14:30.423058 123145507098624 EVENT_LOG_v1 {"time_micros": 1665598470423046, "job": 3, "event": "table_file_deletion", "file_number": 1305}
2022/10/12-18:14:30.427432 4588350976 (Original Log Time 2022/10/12-18:14:30.427425) [db/compaction/compaction_picker_fifo.cc:197] [pikachu] FIFO compaction: nothing to do. Total size 4136, max size 5169
2022/10/12-18:14:30.427448 4588350976 (Original Log Time 2022/10/12-18:14:30.426302) [db/db_impl/db_impl_compaction_flush.cc:1904] [pikachu] Manual compaction starting
2022/10/12-18:20:21.432191 123145508171776 [db/db_impl/db_impl.cc:1103] ------- DUMPING STATS -------
2022/10/12-18:20:21.432271 123145508171776 [db/db_impl/db_impl.cc:1104]
** DB Stats **
Uptime(secs): 351.2 total, 351.2 interval
Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Interval stall: 00:00:0.000 H:M:S, 0.0 percent
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0
Uptime(secs): 351.2 total, 351.2 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
Block cache LRUCache@0x7fd450b040b8#8806 capacity: 8.00 MB usage: 3.78 KB table_size: 256 occupancy: 21 collections: 2 last_copies: 1 last_secs: 0.000143 secs_since: 0
Block cache entry stats(count,size,portion): DataBlock(20,1.99 KB,0.0243187%) Misc(1,0.00 KB,0%)
** Compaction Stats [pikachu] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
L0 4/0 4.04 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
Sum 4/0 4.04 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0
** Compaction Stats [pikachu] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0
Uptime(secs): 351.2 total, 351.2 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
Block cache LRUCache@0x7fd450b040b8#8806 capacity: 8.00 MB usage: 3.78 KB table_size: 256 occupancy: 21 collections: 2 last_copies: 1 last_secs: 0.000143 secs_since: 0
Block cache entry stats(count,size,portion): DataBlock(20,1.99 KB,0.0243187%) Misc(1,0.00 KB,0%)
** File Read Latency Histogram By Level [default] **
** File Read Latency Histogram By Level [pikachu] **
Thanks for the stack-trace, @siying! Originally I was wondering whether it is the case that the test is waiting for compaction but the compaction has already finished so the test ends up waiting forever, and the stack-trace confirms this is exactly the case!
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
* frame #0: 0x00007fff20311cde libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff20344e49 libsystem_pthread.dylib`_pthread_cond_wait + 1298
frame #2: 0x0000000107f66ea8 librocksdb.7.8.0.dylib`rocksdb::port::CondVar::Wait(this=0x00007fd451809270) at port_posix.cc:122:23
frame #3: 0x00000001075ad8d9 librocksdb.7.8.0.dylib`rocksdb::InstrumentedCondVar::WaitInternal(this=0x00007fd451809270) at instrumented_mutex.cc:69:9
frame #4: 0x00000001075ad884 librocksdb.7.8.0.dylib`rocksdb::InstrumentedCondVar::Wait(this=0x00007fd451809270) at instrumented_mutex.cc:62:3
frame #5: 0x0000000106f9104b librocksdb.7.8.0.dylib`rocksdb::DBImpl::WaitForCompact(this=0x00007fd451808800, wait_unscheduled=true) at db_impl_compaction_flush.cc:3852:12
frame #6: 0x0000000107061792 librocksdb.7.8.0.dylib`rocksdb::DBImpl::TEST_WaitForCompact(this=0x00007fd451808800, wait_unscheduled=true) at db_impl_debug.cc:181:10
frame #7: 0x000000010625f95f db_test`rocksdb::DBTest_LevelReopenWithFIFO_Test::TestBody(this=0x00007fd44f017e00) at db_test.cc:596:5
frame #8: 0x00000001064a92e4 db_test`void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(object=0x00007fd44f017e00, method=21 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00, location="the test body")(), char const*) at gtest-all.cc:3899:10
frame #9: 0x00000001064781bb db_test`void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(object=0x00007fd44f017e00, method=21 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00, location="the test body")(), char const*) at gtest-all.cc:3935:14
Has fixed the PR, and all tests pass!
@siying has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@siying The Linter says there's one warning internally at Meta. Do you happen to know what it might be? https://github.com/facebook/rocksdb/pull/10348/checks?check_run_id=8882167321
1 warning internally at Meta.
@siying has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
- Addressed @jay-zhuang's comment.
- Improved HISTORY.md to further describe how the FIFO migration works.
@siying has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
Hi @yhchiang-sol :) I'd like to ask a follow-up question on how does multi-level FIFO interact with file ingestion. Can new files be ingested to non-L0 level under multi-level FIFO?
- My understanding would be "no". If yes, those new files might be compacted to the same level again during CompactRange() which violates the existing assertion here https://github.com/facebook/rocksdb/blob/7.10.fb/db/compaction/compaction_picker_fifo.cc#L435-L436 (unless we plan to change this assertion).
- But then it seems that we can't fulfill "IngestExternalFileOptions.ingest_behind=true" under multi-level FIFO.
