Running with allow WBM stall ended with full hang.

[erez-speedb]

Running the large obj test, 200milion 1K keys never finished.

db_bench flag " -allow_wbm_stalls -db_write_buffer_size=110000000"

Attaching gdb yiled:

#0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55f4c1d73ab4) at ../sysdeps/nptl/futex-internal.h:183 #1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55f4c1d73a60, cond=0x55f4c1d73a88) at pthread_cond_wait.c:508 #2 __pthread_cond_wait (cond=0x55f4c1d73a88, mutex=0x55f4c1d73a60) at pthread_cond_wait.c:647 (More stack frames follow...) (gdb) bt #0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55f4c1d73ab4) at ../sysdeps/nptl/futex-internal.h:183 #1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55f4c1d73a60, cond=0x55f4c1d73a88) at pthread_cond_wait.c:508 #2 __pthread_cond_wait (cond=0x55f4c1d73a88, mutex=0x55f4c1d73a60) at pthread_cond_wait.c:647 #3 0x00007f26c1adae30 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib/x86_64-linux-gnu/libstdc++.so.6 #4 0x000055f4c16d3acf in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x55f4c1d739e0, thread_id=4) at util/threadpool_imp.cc:243 #5 0x000055f4c16d3e13 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x55f4c1dbe530) at util/threadpool_imp.cc:360 #6 0x00007f26c1ae0de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x00007f26c1d1f609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #8 0x00007f26c17cd133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 (gdb) t 3 [Switching to thread 3 (Thread 0x7f26c0ea6700 (LWP 28451))] #0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55f4c1d73ab4) at ../sysdeps/nptl/futex-internal.h:183 183 in ../sysdeps/nptl/futex-internal.h (gdb) bt #0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55f4c1d73ab4) at ../sysdeps/nptl/futex-internal.h:183 #1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55f4c1d73a60, cond=0x55f4c1d73a88) at pthread_cond_wait.c:508 #2 __pthread_cond_wait (cond=0x55f4c1d73a88, mutex=0x55f4c1d73a60) at pthread_cond_wait.c:647 #3 0x00007f26c1adae30 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib/x86_64-linux-gnu/libstdc++.so.6 #4 0x000055f4c16d3acf in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x55f4c1d739e0, thread_id=1) at util/threadpool_imp.cc:243 #5 0x000055f4c16d3e13 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x55f4c1dbefb0) at util/threadpool_imp.cc:360 #6 0x00007f26c1ae0de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x00007f26c1d1f609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #8 0x00007f26c17cd133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 (gdb) t 2 [Switching to thread 2 (Thread 0x7f26c16a7700 (LWP 28450))] #0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55f4c1d73ab4) at ../sysdeps/nptl/futex-internal.h:183 183 in ../sysdeps/nptl/futex-internal.h (gdb) bt #0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x55f4c1d73ab4) at ../sysdeps/nptl/futex-internal.h:183 #1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55f4c1d73a60, cond=0x55f4c1d73a88) at pthread_cond_wait.c:508 #2 __pthread_cond_wait (cond=0x55f4c1d73a88, mutex=0x55f4c1d73a60) at pthread_cond_wait.c:647 #3 0x00007f26c1adae30 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib/x86_64-linux-gnu/libstdc++.so.6 #4 0x000055f4c16d3acf in rocksdb::ThreadPoolImpl::Impl::BGThread (this=0x55f4c1d739e0, thread_id=0) at util/threadpool_imp.cc:243 #5 0x000055f4c16d3e13 in rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper (arg=0x55f4c1dbdd80) at util/threadpool_imp.cc:360 #6 0x00007f26c1ae0de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x00007f26c1d1f609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #8 0x00007f26c17cd133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 (gdb) t 1 [Switching to thread 1 (Thread 0x7f26c16aafc0 (LWP 28449))] #0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffe63171a30) at ../sysdeps/nptl/futex-internal.h:183 183 in ../sysdeps/nptl/futex-internal.h (gdb) bt

[erez-speedb]

2022/12/21-00:05:00.273905 28449 Speedb version: 2.1.0-? (7.7.3) 2022/12/21-00:05:00.273939 28449 Git sha e09733460610c6d6905c633601f1d6f706e15659