Segfaults with lz4mt (threaded mode) and deadlocks with Thread-Pool branch

[blappm]

Hi,

I'm trying to use your extemly fast version of lz4, but from time to time lz4mt is segfaulting due to a issue within threading. It happens with both compression and decompression, but not always at the same place.

We used gcc7 to compile it.

I've also tried the thread-pool variant, but there I see deadlocks - the compression is stalling.

Why haven't you merged the Thread-Pool veriant into the main tree? Was there any reason do not to do it? It seems to me that it would fix the segfault problems.

Best Regards: Martin

[blappm]

Crash with -O2. I'm currently trying to see if I can get crashes with -O0 too.

Core was generated by `lz4mt -1 --numthreads=3 -c'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000000004111f9 in _S_checkstd::__future_base::_State_baseV2 (__p=...) at /usr/include/c++/7/future:552552 if (!static_cast(__p))

(gdb) where

#0 0x00000000004111f9 in _S_checkstd::__future_base::_State_baseV2 (__p=...) at /usr/include/c++/7/future:552#1 wait (this=0x7f37f3bfeff0) at /usr/include/c++/7/future:691#2 (anonymous namespace)::<lambda(int, Lz4Mt::MemPool::Buffer*, int)>::operator()(int, Lz4Mt::MemPool::Buffer , int) const (__closure=0x7f37f506e350, i=262143, srcRawPtr=, srcSize=) at src/lz4mt.cpp:408#3 0x00000000004111f9 in lz4mtCompress (lz4MtContext=, sd=) #4 0x0000000000412072 in __invoke_impl<void, (anonymous namespace)::compress((anonymous namespace)::Ctx&, const (anonymous namespace)::Params&, Lz4Mt::Xxh32&)::<lambda(int, Lz4Mt::MemPool::Buffer, int)>, int, Lz4Mt::MemPool::Buffer*, int> (__f=) at /usr/include/c++/7/bits/invoke.h:60 #5 __invoke<(anonymous namespace)::compress((anonymous namespace)::Ctx&, const (anonymous namespace)::Params&, Lz4Mt::Xxh32&)::<lambda(int, Lz4Mt::MemPool::Buffer*, int)>, int, Lz4Mt::MemPool::Buffer*, int> (__fn=) at /usr/include/c++/7/bits/invoke.h:95 #6 _M_invoke<0, 1, 2, 3> (this=) at /usr/include/c++/7/thread:234#7 operator() (this=) at /usr/include/c++/7/thread:243#8 operator() (this=0x7f37fa1aee30) at /usr/include/c++/7/future:1362#9 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<(anonymous namespace)::compress((anonymous namespace)::Ctx&, const (anonymous namespace)::Params&, Lz4Mt::Xxh32&)::<lambda(int, Lz4Mt::MemPool::Buffer*, int)>, int, Lz4Mt::MemPool::Buffer*, int> >, void> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/7/bits/std_function.h:302 #10 0x000000000040b9eb in operator() (this=) at /usr/include/c++/7/bits/std_function.h:706 #11 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (this=0x7f37f506e300, __f=, __did_set=0x7f37fa1aee07) at /usr/include/c++/7/future:561#12 0x00007f37fc7b8d03 in pthread_once () from /lib64/libpthread.so.0 #13 0x000000000040ec48 in __gthread_once (__func=, __once=0x7f37f506e318) at /usr/include/c++/7/x86_64-suse-linux/bits/gthr-default.h:699 #14 call_once<void (std::__future_base::_State_baseV2::)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>, bool> (__f=<unknown type in /tmp/lz4mt.exe, CU 0x34048, DIE 0x6e524>, __once=...) at /usr/include/c++/7/mutex:684#15 _M_set_result (__ignore_failure=false, __res=..., this=0x7f37f506e300) at /usr/include/c++/7/future:401#16 operator() (__closure=0x7f37f506e0e8) at /usr/include/c++/7/future:1667#17 __invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<(anonymous namespace)::compress((anonymous namespace)::Ctx&, const (anonymous namespace)::Params&, Lz4Mt::Xxh32&)::<lambda(int, Lz4Mt::MemPool::Buffer*, int)>, int, Lz4Mt::MemPool::Buffer*, int> >; _Res = void]::<lambda()> > (__f=) at /usr/include/c++/7/bits/invoke.h:60 #18 __invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<(anonymous namespace)::compress((anonymous namespace)::Ctx&, const (anonymous namespace)::Params&, Lz4Mt::Xxh32&)::<lambda(int, Lz4Mt::MemPool::Buffer*, int)>, int, Lz4Mt::MemPool::Buffer*, int> >; _Res = void]::<lambda()> > (__fn=) at /usr/include/c++/7/bits/invoke.h:95 #19 _M_invoke<0> (this=0x7f37f506e0e8) at /usr/include/c++/7/thread:234#20 operator() (this=0x7f37f506e0e8) at /usr/include/c++/7/thread:243#21 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<(anonymous namespace)::compress((anonymous namespace)::Ctx&, const (anonymous namespace)::Params&, Lz4Mt::Xxh32&)::<lambda(int, Lz4Mt::MemPool::Buffer*, int)>, int, Lz4Mt::MemPool::Buffer*, int> >; _Res = void]::<lambda()> > > >::_M_run(void) (this=0x7f37f506e0e0) at /usr/include/c++/7/thread:186#22 0x0000000000488cdf in std::execute_native_thread_routine (__p=0x7f37f506e0e0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83 #23 0x00007f37fc7b3806 in start_thread () from /lib64/libpthread.so.0 #24 0x00007f37fc2959bd in signalfd () from /lib64/libc.so.6 #25 0x0000000000000000 in ?? () (gdb) p __p $1 = (const std::shared_ptrstd::__future_base::_State_baseV2 &) @0x7f37f3bfeff0: {<std::__shared_ptr<std::__future_base::_State_baseV2, (__gnu_cxx::_Lock_policy)2>> = {<std::__shared_ptr_access<std::__future_base::_State_baseV2, (__gnu_cxx::_Lock_policy)2, false, false>> = {<No data fields>}, _M_ptr = 0x7f37f506e270, _M_refcount = {_M_pi = 0x7f37f506e260}}, <No data fields>}