redpanda icon indicating copy to clipboard operation
redpanda copied to clipboard

CI Failure (exceptional future ignored) in `ConsumerOffsetsConsistencyTest.test_flipping_leadership`

Open rockwotj opened this issue 2 years ago • 27 comments

https://buildkite.com/redpanda/redpanda/builds/36451#018a6738-582b-4a44-96dd-699b359d5375

Module: rptest.tests.consumer_offsets_consistency_test
Class:  ConsumerOffsetsConsistencyTest
Method: test_flipping_leadership
test_id:    rptest.tests.consumer_offsets_consistency_test.ConsumerOffsetsConsistencyTest.test_flipping_leadership
status:     FAIL
run time:   8 minutes 37.887 seconds


    <BadLogLines nodes=docker-rp-15(1) example="WARN  2023-09-05 21:54:57,241 [shard 0:main] seastar - Exceptional future ignored: seastar::no_sharded_instance_exception (sharded instance does not exist: seastar::abort_source), backtrace: 0xedf06 /opt/redpanda_installs/ci/lib/libseastar.so+0x58a53f2 /opt/redpanda_installs/ci/lib/libseastar.so+0x58a4fd5 /opt/redpanda_installs/ci/lib/libseastar.so+0x58a960e /opt/redpanda_installs/ci/lib/libseastar.so+0x58abfef /opt/redpanda_installs/ci/lib/libseastar.so+0x4237e65 0x19adf2 0x19ac84 /opt/redpanda_installs/ci/lib/libv_v_application.so+0x5a45246 /opt/redpanda_installs/ci/lib/libseastar.so+0x423d5bd /opt/redpanda_installs/ci/lib/libseastar.so+0x423db87 /opt/redpanda_installs/ci/lib/libseastar.so+0x45fb744 /opt/redpanda_installs/ci/lib/libseastar.so+0x46089c1 /opt/redpanda_installs/ci/lib/libseastar.so+0x460ec2b /opt/redpanda_installs/ci/lib/libseastar.so+0x460c605 /opt/redpanda_installs/ci/lib/libseastar.so+0x4078ccc /opt/redpanda_installs/ci/lib/libseastar.so+0x40766c6 /opt/redpanda_installs/ci/lib/libv_v_application.so+0x70b6d22 0x17fe74 /opt/redpanda_installs/ci/lib/libc.so.6+0x2d58f /opt/redpanda_installs/ci/lib/libc.so.6+0x2d648 0xa8e64">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 142, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 1237, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-15(1) example="WARN  2023-09-05 21:54:57,241 [shard 0:main] seastar - Exceptional future ignored: seastar::no_sharded_instance_exception (sharded instance does not exist: seastar::abort_source), backtrace: 0xedf06 /opt/redpanda_installs/ci/lib/libseastar.so+0x58a53f2 /opt/redpanda_installs/ci/lib/libseastar.so+0x58a4fd5 /opt/redpanda_installs/ci/lib/libseastar.so+0x58a960e /opt/redpanda_installs/ci/lib/libseastar.so+0x58abfef /opt/redpanda_installs/ci/lib/libseastar.so+0x4237e65 0x19adf2 0x19ac84 /opt/redpanda_installs/ci/lib/libv_v_application.so+0x5a45246 /opt/redpanda_installs/ci/lib/libseastar.so+0x423d5bd /opt/redpanda_installs/ci/lib/libseastar.so+0x423db87 /opt/redpanda_installs/ci/lib/libseastar.so+0x45fb744 /opt/redpanda_installs/ci/lib/libseastar.so+0x46089c1 /opt/redpanda_installs/ci/lib/libseastar.so+0x460ec2b /opt/redpanda_installs/ci/lib/libseastar.so+0x460c605 /opt/redpanda_installs/ci/lib/libseastar.so+0x4078ccc /opt/redpanda_installs/ci/lib/libseastar.so+0x40766c6 /opt/redpanda_installs/ci/lib/libv_v_application.so+0x70b6d22 0x17fe74 /opt/redpanda_installs/ci/lib/libc.so.6+0x2d58f /opt/redpanda_installs/ci/lib/libc.so.6+0x2d648 0xa8e64">

JIRA Link: CORE-1432

rockwotj avatar Sep 06 '23 01:09 rockwotj

This is what I could decode, not exactly helpful... ☹️

0xedf06 
void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:64
seastar::current_backtrace_tasklocal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:98
seastar::current_tasktrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:149
seastar::current_backtrace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/backtrace.cc:182
seastar::report_failed_future(std::exception_ptr const&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/future.cc:216
0x19adf2 
0x19ac84 
[/opt/redpanda_installs/ci/lib/libv_v_application.so+0x5a45246 ](seastar::promise<void>::set_exception(std::exception_ptr&&) at /vectorized/include/seastar/core/future.hh:978)
seastar::parallel_for_each_state::wait_for_one() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/future-util.cc:79
seastar::parallel_for_each_state::run_and_dispose() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/future-util.cc:91
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2750
seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3213
seastar::reactor::do_run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3397
seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3265
seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:276
seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:167
[/opt/redpanda_installs/ci/lib/libv_v_application.so+0x70b6d22 ](application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-085e8b9097d3be57f-1/redpanda/redpanda/src/v/redpanda/application.cc:348)
0x17fe74 
/opt/redpanda_installs/ci/lib/libc.so.6+0x2d58f 
/opt/redpanda_installs/ci/lib/libc.so.6+0x2d648 
0xa8e64

rockwotj avatar Sep 06 '23 01:09 rockwotj

This happens during shutdown, downgrading to sev/low

rockwotj avatar Sep 06 '23 01:09 rockwotj

https://buildkite.com/redpanda/redpanda/builds/36199

rystsov avatar Sep 12 '23 20:09 rystsov

https://buildkite.com/redpanda/redpanda/builds/37716

abhijat avatar Sep 28 '23 08:09 abhijat

https://buildkite.com/redpanda/redpanda/builds/38771#018b21b8-c392-4154-8a8a-e28f1e53cc9e

rockwotj avatar Oct 12 '23 15:10 rockwotj

*https://buildkite.com/redpanda/redpanda/builds/40282

vbotbuildovich avatar Nov 03 '23 16:11 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/41798

vbotbuildovich avatar Dec 13 '23 20:12 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/42006

vbotbuildovich avatar Dec 13 '23 20:12 vbotbuildovich

Assigning to replication as Michal is the only author of the test that is failing.

Note that it appears that the failure mode described in the cover letter isn't the same failure mode as some of these other reported failures.

dotnwat avatar Dec 15 '23 21:12 dotnwat

*https://buildkite.com/redpanda/redpanda/builds/43292

vbotbuildovich avatar Dec 23 '23 00:12 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/43423

vbotbuildovich avatar Jan 09 '24 14:01 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/43796

vbotbuildovich avatar Jan 16 '24 23:01 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/44998

vbotbuildovich avatar Feb 13 '24 20:02 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/45109

vbotbuildovich avatar Feb 20 '24 00:02 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/45181

vbotbuildovich avatar Feb 22 '24 00:02 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/45505

vbotbuildovich avatar Mar 01 '24 00:03 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/12069

vbotbuildovich avatar Mar 04 '24 18:03 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/12141#018e12c9-e355-4a78-a3b1-c03e5779f19d

vbotbuildovich avatar Mar 07 '24 00:03 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/46542

vbotbuildovich avatar Mar 21 '24 21:03 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/46944

vbotbuildovich avatar Mar 29 '24 04:03 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/47094

vbotbuildovich avatar Mar 30 '24 21:03 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/12611

vbotbuildovich avatar Apr 01 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/47526 *https://buildkite.com/redpanda/vtools/builds/12792

vbotbuildovich avatar Apr 10 '24 17:04 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/13280

vbotbuildovich avatar Apr 29 '24 21:04 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/13369

vbotbuildovich avatar May 01 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/48718

vbotbuildovich avatar May 04 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/13739

vbotbuildovich avatar May 15 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/redpanda/builds/49554

vbotbuildovich avatar May 25 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/14509

vbotbuildovich avatar Jun 07 '24 21:06 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/13280 *https://buildkite.com/redpanda/vtools/builds/13369 *https://buildkite.com/redpanda/redpanda/builds/48718 *https://buildkite.com/redpanda/vtools/builds/13739 *https://buildkite.com/redpanda/redpanda/builds/49554 *https://buildkite.com/redpanda/vtools/builds/14509

vbotbuildovich avatar Jun 11 '24 21:06 vbotbuildovich