redpanda
redpanda copied to clipboard
Published
20 hours ago •
redpanda-data
redpanda-data
CI Failure (key symptom) in `RandomNodeOperationsTest.test_node_operations`
https://buildkite.com/redpanda/vtools/builds/13280
Module: rptest.tests.random_node_operations_test
Class: RandomNodeOperationsTest
Method: test_node_operations
Arguments: {
"num_to_upgrade": 0,
"enable_failures": true,
"with_tiered_storage": false
}
test_id: RandomNodeOperationsTest.test_node_operations
status: FAIL
run time: 2765.200 seconds
TimeoutError('')
Traceback (most recent call last):
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
data = self.run_test()
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
return self.test_context.function(self.test)
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 103, in wrapped
r = f(self, *args, **kwargs)
File "/home/ubuntu/redpanda/tests/rptest/tests/random_node_operations_test.py", line 472, in test_node_operations
executor.execute_operation(op)
File "/home/ubuntu/redpanda/tests/rptest/utils/node_operations.py", line 427, in execute_operation
self.recommission(operation.node)
File "/home/ubuntu/redpanda/tests/rptest/utils/node_operations.py", line 366, in recommission
wait_until(recommissioned, timeout_sec=self.timeout, backoff_sec=1)
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 57, in wait_until
raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError
JIRA Link: CORE-2707
*https://buildkite.com/redpanda/vtools/builds/13754 *https://buildkite.com/redpanda/vtools/builds/13923 *https://buildkite.com/redpanda/vtools/builds/13923
PT is lumping different types of errors here, the last one is
<BadLogLines nodes=ip-172-31-15-120(1) example="ERROR 2024-06-26 11:16:21,330 [shard 0:au ] s3 - util.cc:105 - Unexpected error seastar::nested_exception: std::__1::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::__nested<std::__1::system_error> (error generic:32, System error encountered during SSL write: Broken pipe): std::__1::system_error (error system:32, sendmsg: Broken pipe))">
Traceback (most recent call last):
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
data = self.run_test()
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
return self.test_context.function(self.test)
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 190, in wrapped
redpanda.raise_on_bad_logs(
File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1570, in raise_on_bad_logs
lsearcher.search_logs(_searchable_nodes)
File "/home/ubuntu/redpanda/tests/rptest/services/utils.py", line 197, in search_logs
raise BadLogLines(bad_loglines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=ip-172-31-15-120(1) example="ERROR 2024-06-26 11:16:21,330 [shard 0:au ] s3 - util.cc:105 - Unexpected error seastar::nested_exception: std::__1::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::__nested<std::__1::system_error> (error generic:32, System error encountered during SSL write: Broken pipe): std::__1::system_error (error system:32, sendmsg: Broken pipe))">
Closing older-bot-filed CI issues as we transition to a more reliable system.
