redpanda
redpanda copied to clipboard
redpanda-data
CI Failure (Consumed from an unexpected offset) in `PartitionMoveInterruption.test_cancelling_partition_move`
https://buildkite.com/redpanda/redpanda/builds/47713
Module: rptest.tests.partition_move_interruption_test
Class: PartitionMoveInterruption
Method: test_cancelling_partition_move
Arguments: {
"recovery": "restart_recovery",
"compacted": false,
"unclean_abort": true,
"replication_factor": 3
}
test_id: PartitionMoveInterruption.test_cancelling_partition_move
status: FAIL
run time: 141.675 seconds
Exception('VerifiableConsumer-0-139821824201184-worker-1: Traceback (most recent call last):\n File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n self._worker(idx, node)\n File "/root/tests/rptest/services/verifiable_consumer.py", line 356, in _worker\n raise e\n File "/root/tests/rptest/services/verifiable_consumer.py", line 338, in _worker\n handler.handle_records_consumed(event, self.logger)\n File "/root/tests/rptest/services/verifiable_consumer.py", line 101, in handle_records_consumed\n raise AssertionError(msg)\nAssertionError: Consumed from an unexpected offset (1455, 0) for partition TopicPartition(topic=\'topic-zrjtbbdhfp\', partition=0)\n')
Traceback (most recent call last):
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
data = self.run_test()
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
return self.test_context.function(self.test)
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File "/root/tests/rptest/services/cluster.py", line 104, in wrapped
r = f(self, *args, **kwargs)
File "/root/tests/rptest/tests/partition_move_interruption_test.py", line 199, in test_cancelling_partition_move
self.consumer.stop()
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 86, in stop
self._propagate_exceptions()
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 100, in _propagate_exceptions
raise Exception(self.errors)
Exception: VerifiableConsumer-0-139821824201184-worker-1: Traceback (most recent call last):
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/services/background_thread.py", line 38, in _protected_worker
self._worker(idx, node)
File "/root/tests/rptest/services/verifiable_consumer.py", line 356, in _worker
raise e
File "/root/tests/rptest/services/verifiable_consumer.py", line 338, in _worker
handler.handle_records_consumed(event, self.logger)
File "/root/tests/rptest/services/verifiable_consumer.py", line 101, in handle_records_consumed
raise AssertionError(msg)
AssertionError: Consumed from an unexpected offset (1455, 0) for partition TopicPartition(topic='topic-zrjtbbdhfp', partition=0)
JIRA Link: CORE-2353
*https://buildkite.com/redpanda/redpanda/builds/47752 *https://buildkite.com/redpanda/redpanda/builds/47762
This was indirectly caused by https://github.com/redpanda-data/redpanda/pull/17789 that fixed a bug in offset translation of log end offset (and as a result fetch offset validation became stricter). In case of force-abort there is a log discrepancy between leaders and followers that (after a leadership change) leads to offset-out-of-range error and fetch offset reset (previously this wasn't the case because fetch offset validation was incorrect). Although this discrepancy is kind of expected for force-abort, we can minimize the chance of it, see the attached pr.
It seems that this failure popped up in a PR run today:
https://github.com/redpanda-data/redpanda/pull/18105
https://buildkite.com/redpanda/redpanda/builds/48353#018f1bd9-a4db-4853-ad9c-e9b416447aca
https://ci-artifacts.dev.vectorized.cloud/redpanda/48353/018f1bd9-a4db-4853-ad9c-e9b416447aca/vbuild/ducktape/results/final/report.html
*https://buildkite.com/redpanda/redpanda/builds/48470 *https://buildkite.com/redpanda/redpanda/builds/48468
*https://buildkite.com/redpanda/redpanda/builds/48728 *https://buildkite.com/redpanda/redpanda/builds/48729 *https://buildkite.com/redpanda/redpanda/builds/48726
*https://buildkite.com/redpanda/redpanda/builds/48752 *https://buildkite.com/redpanda/redpanda/builds/48760
*https://buildkite.com/redpanda/redpanda/builds/49036 *https://buildkite.com/redpanda/redpanda/builds/49106
*https://buildkite.com/redpanda/redpanda/builds/49295 *https://buildkite.com/redpanda/redpanda/builds/49315 *https://buildkite.com/redpanda/redpanda/builds/49332 *https://buildkite.com/redpanda/redpanda/builds/49356
*https://buildkite.com/redpanda/redpanda/builds/49797 *https://buildkite.com/redpanda/redpanda/builds/49866