redpanda-data
Assertion failures and ASAN violations when SIGINT-ing redpanda during archival
Version & Environment
Debug build of redpanda from tip of dev at the time of writing (0608811e9042e42feff1ccd8e2ff80c981717272). A topic with remote write enabled is present, but no data is being written to it. In my test env, I've set up the AWS bucket to reject all reads, but I don't think that's relevant here.
What went wrong?
Sending a SIGINT signal to redpanda running locally in the scenarion caused above leads to assertion failures or ASAN violations fairly regularly.
Assertion and decoded trace
seastar::output_stream<char>::~output_stream() [CharType = char]: Assertion `!_in_batch && "Was this stream properly closed?"' failed.
/work/dev/vbuild/llvm/src/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:4277
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:754
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:784
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:796
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3662
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3638
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3634
/lib64/libc.so.6+0x3ea2f
/lib64/libc.so.6+0x8ec0b
/lib64/libc.so.6+0x3e985
/lib64/libc.so.6+0x287f3
/lib64/libc.so.6+0x2871a
/lib64/libc.so.6+0x37535
/work/dev/vbuild/debug/clang/rp_deps_install/include/seastar/core/iostream.hh:393
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/tuple:210
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/tuple:384
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/tuple:470
/work/dev/vbuild/debug/clang/rp_deps_install/include/seastar/core/do_with.hh:37
/work/dev/vbuild/debug/clang/rp_deps_install/include/seastar/core/do_with.hh:45
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2329
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2736
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2905
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4139
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/type_traits:3640
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/__functional/invoke.h:61
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:180
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:354
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:507
/work/dev/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:1184
/work/dev/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
/lib64/libc.so.6+0x8cdec
/lib64/libc.so.6+0x11236f
ASAN Violations The output of ASAN is fairly verbose so I've attached it as a file. The redpanda logs precede the ASAN output, so you'll have to scroll to the bottom.
How to reproduce the issue?
I expect that this should reproduce fairly easily with a recent redpanda build with cloud storage enabled.
From the stack traces, I suspect that the recent change in shutdown order might be the cause, but it needs further investigation.
Evgeny mentioned that the ASAN problem is a known issue with the seastar TLS implementation.
