Akumuli
Akumuli copied to clipboard
Published
20 hours ago •
akumuli
akumuli
test_storage deadlocks
I just tried to build master on an uptodate Debian 10 (buster) x64 system. Unfortunately the storage test deadlocks.
strace: Process 85475 attached with 3 threads
[pid 85485] futex(0x55aa70562738, FUTEX_WAIT_PRIVATE, 6, NULL <unfinished ...>
[pid 85486] futex(0x55aa70562738, FUTEX_WAIT_PRIVATE, 6, NULL <unfinished ...>
[pid 85475] futex(0x55aa7056272c, FUTEX_WAIT_PRIVATE, 4294967295, NULL
^C
strace: Process 85475 detached
strace: Process 85485 detached
strace: Process 85486 detached
smurf@dev:~/src/akumuli$ gdb /src/akumuli/obj-x86_64-linux-gnu/unittests/test_storage 85475
GNU gdb (Debian 8.2.1-2) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /src/akumuli/obj-x86_64-linux-gnu/unittests/test_storage...done.
Attaching to program: /src/akumuli/obj-x86_64-linux-gnu/unittests/test_storage, process 85475
[New LWP 85485]
[New LWP 85486]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
futex_wait (private=<optimized out>, expected=4294967295, futex_word=0x55aa7056272c)
at ../sysdeps/unix/sysv/linux/futex-internal.h:61
61 ../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
(gdb) thr 1
[Switching to thread 1 (Thread 0x7f2e8ad13bc0 (LWP 85475))]
#4 0x00007f2e8d0603d9 in std::condition_variable::notify_one() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) whe
#0 futex_wait (private=<optimized out>, expected=4294967295, futex_word=0x55aa7056272c)
at ../sysdeps/unix/sysv/linux/futex-internal.h:61
#1 futex_wait_simple (private=<optimized out>, expected=4294967295, futex_word=0x55aa7056272c)
at ../sysdeps/nptl/futex-internal.h:135
#2 __condvar_quiesce_and_switch_g1 (private=<optimized out>, g1index=<synthetic pointer>, wseq=<optimized out>,
cond=0x55aa70562718) at pthread_cond_common.c:416
#3 __pthread_cond_signal (cond=0x55aa70562718) at pthread_cond_signal.c:78
#4 0x00007f2e8d0603d9 in std::condition_variable::notify_one() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x000055aa6df31079 in Akumuli::MetadataStorage::add_rescue_point (this=0x55aa70562690, id=<optimized out>, val=...)
at ./libakumuli/metadatastorage.cpp:391
#6 0x000055aa6df1629b in Akumuli::Storage::_update_rescue_points (rpoints=..., id=<optimized out>,
this=<optimized out>) at /usr/include/c++/8/bits/shared_ptr_base.h:1018
#7 Akumuli::StorageSession::write (this=0x55aa6e3e28b0, sample=...) at ./libakumuli/storage2.cpp:159
#8 0x000055aa6deea93f in fill_data (
session=std::shared_ptr<class Akumuli::StorageSession> (use count 2, weak count 1) = {...}, begin=<optimized out>,
end=11000, names=std::vector of length 100, capacity 128 = {...}) at /usr/include/c++/8/bits/shared_ptr_base.h:1018
#9 0x000055aa6df05b7f in test_wal_recovery (cardinality=100, begin=1000, end=11000)
at /usr/include/c++/8/ext/atomicity.h:96
#10 0x000055aa6df06d9c in Test_wal_recovery_1::test_method (this=<synthetic pointer>)
at ./unittests/test_storage.cpp:1688
#11 Test_wal_recovery_1_invoker () at ./unittests/test_storage.cpp:1687
#12 0x00007f2e8d3b8f5e in boost::detail::function::function_obj_invoker0<boost::detail::forward, int>::invoke(boost::detail::function::function_buffer&) () from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.67.0
#13 0x00007f2e8d3b8b65 in boost::execution_monitor::catch_signals(boost::function<int ()> const&) ()
from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.67.0
#14 0x00007f2e8d3b8c31 in boost::execution_monitor::execute(boost::function<int ()> const&) ()
from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.67.0
#15 0x00007f2e8d3b8cfd in boost::execution_monitor::vexecute(boost::function<void ()> const&) ()
from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.67.0
#16 0x00007f2e8d3e08a9 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned int) () from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.67.0
#17 0x00007f2e8d3bb46b in ?? () from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.67.0
#18 0x00007f2e8d3bb72f in ?? () from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.67.0
#19 0x00007f2e8d3bdfd5 in boost::unit_test::framework::run(unsigned long, bool) ()
from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.67.0
#20 0x00007f2e8d3de318 in boost::unit_test::unit_test_main(bool (*)(), int, char**) ()
from /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.67.0
#21 0x00007f2e8cc7009b in __libc_start_main (main=0x55aa6dee7890 <main(int, char**)>, argc=1, argv=0x7ffcb5f6acd8,
init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcb5f6acc8)
at ../csu/libc-start.c:308
#22 0x000055aa6deea2ea in _start () at ./unittests/test_storage.cpp:1758
(gdb) thr 2
[Switching to thread 2 (Thread 0x7f2e8ad08700 (LWP 85485))]
#0 futex_wait (private=0, expected=<optimized out>, futex_word=0x55aa70562738)
at ../sysdeps/unix/sysv/linux/futex-internal.h:61
61 ../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
(gdb) whe
#0 futex_wait (private=0, expected=<optimized out>, futex_word=0x55aa70562738)
at ../sysdeps/unix/sysv/linux/futex-internal.h:61
#1 futex_wait_simple (private=0, expected=<optimized out>, futex_word=0x55aa70562738)
at ../sysdeps/nptl/futex-internal.h:135
#2 __condvar_acquire_lock (private=0, cond=0x55aa70562718) at pthread_cond_common.c:280
#3 __condvar_cancel_waiting (cond=0x55aa70562718, seq=14, g=<optimized out>, private=0) at pthread_cond_wait.c:86
#4 0x00007f2e8d295497 in __pthread_cond_wait_common (abstime=0x7f2e8ad07d50, mutex=0x55aa705626c8,
cond=0x55aa70562718) at pthread_cond_wait.c:554
#5 __pthread_cond_timedwait (cond=0x55aa70562718, mutex=0x55aa705626c8, abstime=0x7f2e8ad07d50)
at pthread_cond_wait.c:667
#6 0x000055aa6df2ac27 in __gthread_cond_timedwait (__abs_timeout=0x7f2e8ad07d50, __mutex=0x55aa705626c8,
__cond=0x55aa70562718) at /usr/include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:871
#7 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (
__atime=..., __lock=<synthetic pointer>..., this=0x55aa70562718) at /usr/include/c++/8/condition_variable:178
#8 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=...,
__lock=<synthetic pointer>..., this=0x55aa70562718) at /usr/include/c++/8/condition_variable:106
#9 std::condition_variable::wait_for<long, std::ratio<1l, 1000000l> > (__rtime=..., __lock=<synthetic pointer>...,
this=0x55aa70562718) at /usr/include/c++/8/condition_variable:143
#10 Akumuli::MetadataStorage::wait_for_sync_request (this=0x55aa70562690, timeout_us=timeout_us@entry=10000)
at ./libakumuli/metadatastorage.cpp:381
#11 0x000055aa6df1af25 in Akumuli::Storage::<lambda()>::operator() (__closure=0x55aa7053dcf8)
at /usr/include/c++/8/bits/shared_ptr_base.h:1018
#12 std::__invoke_impl<void, Akumuli::Storage::start_sync_worker()::<lambda()> > (__f=...)
at /usr/include/c++/8/bits/invoke.h:60
#13 std::__invoke<Akumuli::Storage::start_sync_worker()::<lambda()> > (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#14 std::thread::_Invoker<std::tuple<Akumuli::Storage::start_sync_worker()::<lambda()> > >::_M_invoke<0> (
this=0x55aa7053dcf8) at /usr/include/c++/8/thread:244
#15 std::thread::_Invoker<std::tuple<Akumuli::Storage::start_sync_worker()::<lambda()> > >::operator() (
this=0x55aa7053dcf8) at /usr/include/c++/8/thread:253
#16 std::thread::_State_impl<std::thread::_Invoker<std::tuple<Akumuli::Storage::start_sync_worker()::<lambda()> > > >::_M_run(void) (this=0x55aa7053dcf0) at /usr/include/c++/8/thread:196
#17 0x00007f2e8d065b2f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#18 0x00007f2e8d28efa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#19 0x00007f2e8cd454cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) thr 3
[Switching to thread 3 (Thread 0x7f2e8a48f700 (LWP 85486))]
#0 futex_wait (private=0, expected=<optimized out>, futex_word=0x55aa70562738)
at ../sysdeps/unix/sysv/linux/futex-internal.h:61
61 in ../sysdeps/unix/sysv/linux/futex-internal.h
(gdb) whe
#0 futex_wait (private=0, expected=<optimized out>, futex_word=0x55aa70562738)
at ../sysdeps/unix/sysv/linux/futex-internal.h:61
#1 futex_wait_simple (private=0, expected=<optimized out>, futex_word=0x55aa70562738)
at ../sysdeps/nptl/futex-internal.h:135
#2 __condvar_acquire_lock (private=0, cond=0x55aa70562718) at pthread_cond_common.c:280
#3 __condvar_cancel_waiting (cond=0x55aa70562718, seq=15, g=<optimized out>, private=0) at pthread_cond_wait.c:86
#4 0x00007f2e8d295497 in __pthread_cond_wait_common (abstime=0x7f2e8a48ed50, mutex=0x55aa705626c8,
cond=0x55aa70562718) at pthread_cond_wait.c:554
#5 __pthread_cond_timedwait (cond=0x55aa70562718, mutex=0x55aa705626c8, abstime=0x7f2e8a48ed50)
at pthread_cond_wait.c:667
#6 0x000055aa6df2ac27 in __gthread_cond_timedwait (__abs_timeout=0x7f2e8a48ed50, __mutex=0x55aa705626c8,
__cond=0x55aa70562718) at /usr/include/x86_64-linux-gnu/c++/8/bits/gthr-default.h:871
#7 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (
__atime=..., __lock=<synthetic pointer>..., this=0x55aa70562718) at /usr/include/c++/8/condition_variable:178
#8 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=...,
__lock=<synthetic pointer>..., this=0x55aa70562718) at /usr/include/c++/8/condition_variable:106
#9 std::condition_variable::wait_for<long, std::ratio<1l, 1000000l> > (__rtime=..., __lock=<synthetic pointer>...,
this=0x55aa70562718) at /usr/include/c++/8/condition_variable:143
#10 Akumuli::MetadataStorage::wait_for_sync_request (this=0x55aa70562690, timeout_us=timeout_us@entry=10000)
at ./libakumuli/metadatastorage.cpp:381
#11 0x000055aa6df1af25 in Akumuli::Storage::<lambda()>::operator() (__closure=0x55aa705053d8)
at /usr/include/c++/8/bits/shared_ptr_base.h:1018
#12 std::__invoke_impl<void, Akumuli::Storage::start_sync_worker()::<lambda()> > (__f=...)
at /usr/include/c++/8/bits/invoke.h:60
#13 std::__invoke<Akumuli::Storage::start_sync_worker()::<lambda()> > (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#14 std::thread::_Invoker<std::tuple<Akumuli::Storage::start_sync_worker()::<lambda()> > >::_M_invoke<0> (
this=0x55aa705053d8) at /usr/include/c++/8/thread:244
#15 std::thread::_Invoker<std::tuple<Akumuli::Storage::start_sync_worker()::<lambda()> > >::operator() (
this=0x55aa705053d8) at /usr/include/c++/8/thread:253
#16 std::thread::_State_impl<std::thread::_Invoker<std::tuple<Akumuli::Storage::start_sync_worker()::<lambda()> > > >::_M_run(void) (this=0x55aa705053d0) at /usr/include/c++/8/thread:196
#17 0x00007f2e8d065b2f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#18 0x00007f2e8d28efa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#19 0x00007f2e8cd454cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
