CI Failure (Segmentation fault) in `RandomNodeOperationsTest.test_node_operations`

[vbotbuildovich]

https://buildkite.com/redpanda/redpanda/builds/47431

Module: rptest.tests.random_node_operations_test
Class: RandomNodeOperationsTest
Method: test_node_operations
Arguments: {
    "num_to_upgrade": 0,
    "enable_failures": true,
    "with_tiered_storage": true
}

test_id:    RandomNodeOperationsTest.test_node_operations
status:     FAIL
run time:   520.485 seconds

<BadLogLines nodes=docker-rp-13(1) example="Segmentation fault on shard 0.">
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 177, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 1503, in raise_on_bad_logs
    lsearcher.search_logs(_searchable_nodes)
  File "/root/tests/rptest/services/utils.py", line 156, in search_logs
    raise BadLogLines(bad_loglines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-13(1) example="Segmentation fault on shard 0.">

JIRA Link: CORE-2265

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12861

[mmaslankaprv]

related with: #13301

[dotnwat]

Segmentation fault on shard 0.
Backtrace:
  0x7e9bf23
  0x7ee0bd3
  linux-vdso.so.1+0x81f
  0x5f27dbb
  0x5f20007
  0x2dc0f3f
  0x7e9684b
  0x7e98eff
  0x7e970d3
  0x7dbc32b
  0x7dbae7f
  0x2cababb
  0x812b83b
  /opt/redpanda_installs/ci/lib/libc.so.6+0x2b1c7
  /opt/redpanda_installs/ci/lib/libc.so.6+0x2b29f
  0x2ca49ef

Backtrace:
[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::'lambda'(seastar::frame)>(seastar::backtrace_buffer::append_backtrace()::'lambda'(seastar::frame)&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:824
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:854
 (inlined by) seastar::print_with_backtrace(char const*, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:866
seastar::sigsegv_action() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4150
 (inlined by) void seastar::install_oneshot_signal_handler<11, (void (*)())(&seastar::sigsegv_action())>()::'lambda'(int, siginfo_t*, void*)::operator()(int, siginfo_t*, void*) const at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4131
 (inlined by) void seastar::install_oneshot_signal_handler<11, (void (*)())(&seastar::sigsegv_action())>()::'lambda'(int, siginfo_t*, void*)::__invoke(int, siginfo_t*, void*) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4127
/opt/homebrew/opt/llvm/bin/llvm-addr2line: error: 'linux-vdso.so.1': No such file or directory
linux-vdso.so.1 0x81f
absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::FlatHashSetPolicy<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, absl::lts_20230802::hash_internal::Hash<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>>::prefetch_heap_block() const at /vectorized/include/absl/container/internal/raw_hash_set.h:2735
 (inlined by) absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::FlatHashSetPolicy<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, absl::lts_20230802::hash_internal::Hash<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>>::const_iterator absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::FlatHashSetPolicy<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, absl::lts_20230802::hash_internal::Hash<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>>::find<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>> const&) const at /vectorized/include/absl/container/internal/raw_hash_set.h:2361
 (inlined by) bool absl::lts_20230802::container_internal::raw_hash_set<absl::lts_20230802::container_internal::FlatHashSetPolicy<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, absl::lts_20230802::hash_internal::Hash<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>>::contains<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>> const&) const at /vectorized/include/absl/container/internal/raw_hash_set.h:2367
 (inlined by) cluster::has_quorum(absl::lts_20230802::flat_hash_set<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, absl::lts_20230802::hash_internal::Hash<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>> const&, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard>> const&)::$_0::operator()(model::broker_shard const&) const at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0356393afa1e7565b-1/redpanda/redpanda/src/v/cluster/partition_balancer_planner.cc:545
 (inlined by) std::__1::iterator_traits<std::__1::__wrap_iter<model::broker_shard const*>>::difference_type std::__1::count_if[abi:v160004]<std::__1::__wrap_iter<model::broker_shard const*>, cluster::has_quorum(absl::lts_20230802::flat_hash_set<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, absl::lts_20230802::hash_internal::Hash<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>> const&, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard>> const&)::$_0>(std::__1::__wrap_iter<model::broker_shard const*>, std::__1::__wrap_iter<model::broker_shard const*>, cluster::has_quorum(absl::lts_20230802::flat_hash_set<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, absl::lts_20230802::hash_internal::Hash<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>> const&, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard>> const&)::$_0) at /vectorized/llvm/bin/../include/c++/v1/__algorithm/count_if.h:28
 (inlined by) cluster::has_quorum(absl::lts_20230802::flat_hash_set<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>, absl::lts_20230802::hash_internal::Hash<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::equal_to<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>, std::__1::allocator<detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true>>>> const&, std::__1::vector<model::broker_shard, std::__1::allocator<model::broker_shard>> const&) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0356393afa1e7565b-1/redpanda/redpanda/src/v/cluster/partition_balancer_planner.cc:541
 (inlined by) auto cluster::partition_balancer_planner::request_context::do_with_partition<seastar::noncopyable_function<seastar::bool_class<seastar::stop_iteration_tag> (cluster::partition_balancer_planner::partition&)>>(model::ntp const&, cluster::partition_assignment const&, seastar::noncopyable_function<seastar::bool_class<seastar::stop_iteration_tag> (cluster::partition_balancer_planner::partition&)>&) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0356393afa1e7565b-1/redpanda/redpanda/src/v/cluster/partition_balancer_planner.cc:981
cluster::partition_balancer_planner::request_context::for_each_partition_random_order(seastar::noncopyable_function<seastar::bool_class<seastar::stop_iteration_tag> (cluster::partition_balancer_planner::partition&)>) (.resume) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0356393afa1e7565b-1/redpanda/redpanda/src/v/cluster/partition_balancer_planner.cc:1063
std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<void>::promise_type>::resume[abi:v160004]() const at /vectorized/llvm/bin/../include/c++/v1/__coroutine/coroutine_handle.h:169
 (inlined by) seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose() at /vectorized/include/seastar/core/coroutine.hh:125
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3265
seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:276
seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:167
application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0356393afa1e7565b-1/redpanda/redpanda/src/v/redpanda/application.cc:421
main at /var/lib/buildkite-agent/builds/buildkite-arm64-builders-i-0356393afa1e7565b-1/redpanda/redpanda/src/v/redpanda/main.cc:22
/opt/homebrew/opt/llvm/bin/llvm-addr2line: error: '/opt/redpanda_installs/ci/lib/libc.so.6': No such file or directory
/opt/redpanda_installs/ci/lib/libc.so.6 0x2b1c7
/opt/redpanda_installs/ci/lib/libc.so.6 0x2b29f
_start at ??:0

[mmaslankaprv]

Related issues: https://github.com/redpanda-data/redpanda/issues/17751 https://github.com/redpanda-data/redpanda/issues/16510 https://github.com/redpanda-data/redpanda/issues/16533 https://github.com/redpanda-data/redpanda/issues/13301