crash when upgrading from v21.11.19 to v22.1.7

[andrwng]

Version & Environment

Redpanda version: (use rpk version): v21.11.19 => v22.1.7

I have a test in flight (branch here) that runs through two upgrades, one from v21.11.19 => v22.1.7 => v22.2.2 (roughly, dev), all the while running a produce and a few consumer workloads with FranzGoVerifiable services.

What went wrong?

A node crashed while all nodes were on v22.1.7. Snippet:

Backtrace:                                                                                           
[Backtrace #0]                                                                                       
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:760
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:790
 (inlined by) seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:802
 (inlined by) seastar::sigsegv_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3716
 (inlined by) operator() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3702
 (inlined by) __invoke at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3698  
?? ??:0                                                                                              
model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >) at ??:?
 (inlined by) std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>::valueless_by_exception() const at /vectorized/llvm/bin/../include/c++/v1/variant:739
 (inlined by) std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t>::valueless_by_exception() const at /vectorized/llvm/bin/../include/c++/v1/variant:1390
 (inlined by) void std::__1::__throw_if_valueless<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t> const&>(std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t> const&) at /vectorized/llvm/bin/../include/c++/v1/variant:1642
 (inlined by) decltype(auto) std::__1::visit<seastar::internal::variant_visitor<model::record_batch_reader::impl::is_slice_empty() const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> > const&)#1}, model::record_batch_reader::impl::is_slice_empty() const::{lambda(model::record_batch_reader::foreign_data_t const&)#1}>, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t> const&>(seastar::internal::variant_visitor<model::record_batch_reader::impl::is_slice_empty() const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> > const&)#1}, model::record_batch_reader::impl::is_slice_empty() const::{lambda(model::record_batch_reader::foreign_data_t const&)#1}>&&, std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t> const&) at /vectorized/llvm/bin/../include/c++/v1/variant:1653
 (inlined by) auto seastar::visit<std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t> const&, model::record_batch_reader::impl::is_slice_empty() const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> > const&)#1}, model::record_batch_reader::impl::is_slice_empty() const::{lambda(model::record_batch_reader::foreign_data_t const&)#1}>(std::__1::variant<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> >, model::record_batch_reader::foreign_data_t> const&, model::record_batch_reader::impl::is_slice_empty() const::{lambda(seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> > const&)#1}&&, model::record_batch_reader::impl::is_slice_empty() const::{lambda(model::record_batch_reader::foreign_data_t const&)#1}&&) at /vectorized/include/seastar/util/variant_utils.hh:71
 (inlined by) model::record_batch_reader::impl::is_slice_empty() const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-04c0dca86ffe169ff-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:74
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-04c0dca86ffe169ff-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:155
 (inlined by) seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<model::record_batch_reader::impl::do_action<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, model::record_batch_reader::impl::do_consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >, model::record_batch_reader::impl::do_consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&&)::{lambda()#1}&>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) seastar::future<void> seastar::repeat<model::record_batch_reader::impl::do_action<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, model::record_batch_reader::impl::do_consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >, model::record_batch_reader::impl::do_consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&&)::{lambda()#1}>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&&) at /vectorized/include/seastar/core/loop.hh:118
 (inlined by) auto model::record_batch_reader::impl::do_action<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, model::record_batch_reader::impl::do_consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >, model::record_batch_reader::impl::do_consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&&) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-04c0dca86ffe169ff-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:151
 (inlined by) auto model::record_batch_reader::impl::do_consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-04c0dca86ffe169ff-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:142
 (inlined by) operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-04c0dca86ffe169ff-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:99
 (inlined by) decltype ((std::__1::forward<model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&>({parm#1}))(std::__1::forward<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&>({parm#2}))) std::__1::__invoke_constexpr<model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&, model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&>(model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&, model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3700
 (inlined by) decltype(auto) std::__1::__apply_tuple_impl<model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&, std::__1::tuple<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>&, 0ul>(model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&, std::__1::tuple<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>&, std::__1::__tuple_indices<0ul>) at /vectorized/llvm/bin/../include/c++/v1/tuple:1415
 (inlined by) decltype(auto) std::__1::apply<model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&, std::__1::tuple<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>&>(model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&, std::__1::tuple<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>&) at /vectorized/llvm/bin/../include/c++/v1/tuple:1424
 (inlined by) auto seastar::internal::do_with_impl<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&&, model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&&) at /vectorized/include/seastar/core/do_with.hh:96
 (inlined by) seastar::future<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> > > seastar::futurize<seastar::future<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> > > >::invoke<seastar::future<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> > > (*&)(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&&, model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&&), model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}>(seastar::future<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> > > (*&)(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&&, model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&&), model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&&, model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) auto seastar::futurize_invoke<seastar::future<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> > > (*&)(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&&, model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&&), model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}>(seastar::future<seastar::circular_buffer<model::record_batch, std::__1::allocator<model::record_batch> > > (*&)(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&&, model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&&), model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&&, model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&&) at /vectorized/include/seastar/core/future.hh:2166
 (inlined by) auto seastar::do_with<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&&, model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::{lambda(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer&)#1}&&) at /vectorized/include/seastar/core/do_with.hh:131
 (inlined by) auto model::record_batch_reader::impl::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-04c0dca86ffe169ff-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:97
 (inlined by) auto model::record_batch_reader::consume<model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer>(model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >)::memory_batch_consumer, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >) && at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-04c0dca86ffe169ff-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.h:247
 (inlined by) model::consume_reader_to_memory(model::record_batch_reader, std::__1::chrono::time_point<seastar::lowres_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> > >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-04c0dca86ffe169ff-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/model/record_batch_reader.cc:178
raft::details::share_reader(model::record_batch_reader, unsigned long, bool) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-04c0dca86ffe169ff-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/raft/consensus_utils.cc:102
raft::details::foreign_share_n(model::record_batch_reader&&, unsigned long) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-04c0dca86ffe169ff-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/raft/consensus_utils.cc:125
operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-04c0dca86ffe169ff-1/redpanda/redpanda/vbuild/release/clang/../../../src/v/raft/replicate_entries_stm.cc:36
 (inlined by) seastar::future<raft::append_entries_request> seastar::futurize<seastar::future<raft::append_entries_request> >::invoke<raft::replicate_entries_stm::share_request()::$_2>(raft::replicate_entries_stm::share_request()::$_2&&) at /vectorized/include/seastar/core/future.hh:2135
 (inlined by) auto seastar::futurize_invoke<raft::replicate_entries_stm::share_request()::$_2>(raft::replicate_entries_stm::share_request()::$_2&&) at /vectorized/include/seastar/core/future.hh:2166
 (inlined by) operator()<seastar::semaphore_units<> > at /vectorized/include/seastar/core/semaphore.hh:554
decltype ((std::__1::forward<seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}&>({parm#1}))(std::__1::forward<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >({parm#2}))) std::__1::__invoke<seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >(seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3694
 (inlined by) std::__1::invoke_result<seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::type std::__1::invoke<seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >(seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&&) at /vectorized/llvm/bin/../include/c++/v1/functional:2989
 (inlined by) auto seastar::internal::future_invoke<seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >(seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}&, raft::replicate_entries_stm::share_request()::$_2&&) at /vectorized/include/seastar/core/future.hh:1211
 (inlined by) operator() at /vectorized/include/seastar/core/future.hh:1582                          
 (inlined by) void seastar::futurize<seastar::future<raft::append_entries_request> >::satisfy_with_result_of<seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then_impl_nrvo<seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}, seastar::future<raft::append_entries_request> >(seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}&&)::{lambda(seastar::internal::promise_base_with_type<raft::append_entries_request>&&, seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<auto:1, auto:3>&, unsigned long, auto:2&&)::{lambda(auto:1)#1}&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >&&)#1}::operator()(seastar::internal::promise_base_with_type<raft::append_entries_request>&&, seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<raft::append_entries_request>&&, seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}&&) at /vectorized/include/seastar/core/future.hh:2120
 (inlined by) operator() at /vectorized/include/seastar/core/future.hh:1575                          
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<raft::append_entries_request>, seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}, seastar::future<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::then_impl_nrvo<seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}, seastar::future<raft::append_entries_request> >(seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock>&, unsigned long, raft::replicate_entries_stm::share_request()::$_2&&)::{lambda(auto:1)#1}&&)::{lambda(seastar::internal::promise_base_with_type<raft::append_entries_request>&&, seastar::with_semaphore<seastar::semaphore_default_exception_factory, raft::replicate_entries_stm::share_request()::$_2, std::__1::chrono::steady_clock>(seastar::basic_semaphore<auto:1, auto:3>&, unsigned long, auto:2&&)::{lambda(auto:1)#1}&, seastar::future_state<seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >&&)#1}, seastar::semaphore_units<seastar::semaphore_default_exception_factory, std::__1::chrono::steady_clock> >::run_and_dispose() at /vectorized/include/seastar/core/future.hh:767
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2378
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2787
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2956
operator() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4171              
 (inlined by) decltype ((std::__1::forward<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_93&>({parm#1}))()) std::__1::__invoke<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_93&>(seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_93&) at /vectorized/llvm/bin/../include/c++/v1/type_traits:3694
 (inlined by) void std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_93&>(seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_93&) at /vectorized/llvm/bin/../include/c++/v1/__functional_base:348
 (inlined by) std::__1::__function::__alloc_func<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_93, std::__1::allocator<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_93>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/functional:1558
 (inlined by) std::__1::__function::__func<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_93, std::__1::allocator<seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::$_93>, void ()>::operator()() at /vectorized/llvm/bin/../include/c++/v1/functional:1732
std::__1::__function::__value_func<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/functional:1885
 (inlined by) std::__1::function<void ()>::operator()() const at /vectorized/llvm/bin/../include/c++/v1/functional:2560
 (inlined by) seastar::posix_thread::start_routine(void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:60
/opt/redpanda/lib/libpthread.so.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=7b0cdaf878ab4f99078439d864af70a5fd7b5a2c, for GNU/Linux 3.2.0, not stripped
                                                                                                     
start_thread at pthread_create.c:?                                                                   
/opt/redpanda/lib/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6e3c087aca9b39549e4ba92c451f1e399b586e28, for GNU/Linux 3.2.0, not stripped

Full test logs b2b-upgrades.tar.gz

[jcsp]

Observations:

• Crash is happening several minutes after startup of 22.1.7 on docker-rp-2, so unlikely to be an issue replaying/decoding the content that an earlier version wrote (this is all long after log replay has happened)
• It's in a reader belonging to raft (replicate_entries_stm), in the Suggests that perhaps there is a lifetime issue with these readers vs. the lifetime of the stm or the readers within it.
• Calls to share_request are more common when the node is having to retry internal RPCs to raft peers, so level of system stress could contribute to chance of reproducing

[jcsp]

I don't see us fixing any raft crashes the src/v/raft commit history 22.1->22.2, so it may well be that this issue is still present and we only saw it in 22.1 by coincindence. No indication that it's a regression in 22.2.

A test to hunt for this particular issues might be to run the same verifier workload (especially the random reads, which will much increase the probability of displacing recent data from cache and thereby making a replicate reader experience enough walltime to hit a race case), and just do repeated rolling restarts and/or SIGSTOP/SIGSTARTs on peers so that replication has to do restarts.

[mmaslankaprv]

may it be related to the bad_allocs we are seeing in logs ?

[andrwng]

Ah maybe this is an environmental issue then, this crash came in on local ducktape and this is deemed a scale test, in which case maybe there's no expectation of this working well. I think that would explain this given the backtrace is showing up in the __alloc_func

[andrwng]

I've run this test several dozens of times with beefier nodes with CDT and have not seen these crashes.