redpanda
redpanda copied to clipboard
Published
20 hours ago •
redpanda-data
redpanda-data
CI Failure (Unknown topic or partition) in `CreateTopicUpgradeTest.test_retention_config_on_upgrade_from_v22_2_to_v22_3`
https://buildkite.com/redpanda/vtools/builds/11635
Module: rptest.tests.topic_creation_test
Class: CreateTopicUpgradeTest
Method: test_retention_config_on_upgrade_from_v22_2_to_v22_3
Arguments: {
"cloud_storage_type": 1
}
test_id: CreateTopicUpgradeTest.test_retention_config_on_upgrade_from_v22_2_to_v22_3
status: FAIL
run time: 69.729 seconds
RpkException('command /opt/redpanda/bin/rpk topic -X brokers=rp-node-17-barely-engaging-crane:9092,rp-node-8-barely-engaging-crane:9092,rp-node-22-barely-engaging-crane:9092 produce --key key -z none --delivery-timeout 5s -f %v test-topic-post-upgrade-si -v returned 1, output: ', '12:29:15.224 INFO flushing\n12:29:15.225 INFO producing to a new topic for the first time, fetching metadata to learn its partitions {"topic": "test-topic-post-upgrade-si"}\n12:29:15.225 INFO immediate metadata update triggered {"why": "forced load because we are producing to a topic for the first time"}\n12:29:15.225 DEBUG opening connection to broker {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "seed_0"}\n12:29:15.226 DEBUG connection opened to broker {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "seed_0"}\n12:29:15.226 DEBUG issuing api versions request {"broker": "seed_0", "version": 3}\n12:29:15.226 DEBUG wrote ApiVersions v3 {"broker": "seed_0", "bytes_written": 31, "write_wait": "42.398µs", "time_to_write": "19.215µs", "err": null}\n12:29:15.226 DEBUG read ApiVersions v3 {"broker": "seed_0", "bytes_read": 247, "read_wait": "37.133µs", "time_to_read": "292.958µs", "err": null}\n12:29:15.226 DEBUG connection initialized successfully {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "seed_0"}\n12:29:15.226 DEBUG wrote Metadata v7 {"broker": "seed_0", "bytes_written": 50, "write_wait": "1.68255ms", "time_to_write": "17.069µs", "err": null}\n12:29:15.227 DEBUG read Metadata v7 {"broker": "seed_0", "bytes_read": 271, "read_wait": "35.189µs", "time_to_read": "221.844µs", "err": null}\n12:29:15.227 INFO done waiting for metadata for new topic {"topic": "test-topic-post-upgrade-si"}\n12:29:15.227 INFO initializing producer id\n12:29:15.227 DEBUG opening connection to broker {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "3"}\n12:29:15.227 DEBUG connection opened to broker {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "3"}\n12:29:15.227 DEBUG issuing api versions request {"broker": "3", "version": 3}\n12:29:15.227 DEBUG wrote ApiVersions v3 {"broker": "3", "bytes_written": 31, "write_wait": "6.763µs", "time_to_write": "14.027µs", "err": null}\n12:29:15.228 DEBUG read ApiVersions v3 {"broker": "3", "bytes_read": 247, "read_wait": "18.149µs", "time_to_read": "327.305µs", "err": null}\n12:29:15.228 DEBUG connection initialized successfully {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "3"}\n12:29:15.228 DEBUG wrote InitProducerID v3 {"broker": "3", "bytes_written": 34, "write_wait": "871.293µs", "time_to_write": "8.515µs", "err": null}\n12:29:15.229 DEBUG read InitProducerID v3 {"broker": "3", "bytes_read": 26, "read_wait": "30.908µs", "time_to_read": "1.477385ms", "err": null}\n12:29:15.229 INFO producer id initialization success {"id": 1001, "epoch": 0}\n12:29:15.229 DEBUG opening connection to broker {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "3"}\n12:29:15.230 DEBUG connection opened to broker {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "3"}\n12:29:15.230 DEBUG connection initialized successfully {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "3"}\n12:29:15.230 DEBUG wrote Produce v7 {"broker": "3", "bytes_written": 131216, "write_wait": "467.908µs", "time_to_write": "109.659µs", "err": null}\n12:29:15.230 DEBUG read Produce v7 {"broker": "3", "bytes_read": 78, "read_wait": "39.859µs", "time_to_read": "328.073µs", "err": null}\n12:29:15.230 DEBUG retry batches processed {"wanted_metadata_update": true, "triggering_metadata_update": false, "should_backoff": true}\n12:29:15.230 DEBUG produced {"broker": "3", "to": "test-topic-post-upgrade-si[0{retrying@-1,1(UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.)}]"}\n12:29:15.530 DEBUG wrote Produce v7 {"broker": "3", "bytes_written": 131216, "write_wait": "62.625µs", "time_to_write": "63.885µs", "err": null}\n12:29:15.530 DEBUG read Produce v7 {"broker": "3", "bytes_read": 78, "read_wait": "94.542µs", "time_to_read": "420.064µs", "err": null}\n12:29:15.530 DEBUG retry batches processed {"wanted_metadata_update": true, "triggering_metadata_update": true, "should_backoff": false}\n12:29:15.530 DEBUG produced {"broker": "3", "to": "test-topic-post-upgrade-si[0{retrying@-1,1(UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.)}]"}\n12:29:15.530 INFO metadata update triggered {"why": "produce request had retry batches"}\n12:29:15.530 DEBUG opening connection to broker {"addr": "rp-node-22-barely-engaging-crane:9092", "broker": "2"}\n12:29:15.531 DEBUG connection opened to broker {"addr": "rp-node-22-barely-engaging-crane:9092", "broker": "2"}\n12:29:15.531 DEBUG issuing api versions request {"broker": "2", "version": 3}\n12:29:15.531 DEBUG wrote ApiVersions v3 {"broker": "2", "bytes_written": 31, "write_wait": "7.797µs", "time_to_write": "9.188µs", "err": null}\n12:29:15.531 DEBUG read ApiVersions v3 {"broker": "2", "bytes_read": 247, "read_wait": "38.529µs", "time_to_read": "133.086µs", "err": null}\n12:29:15.531 DEBUG connection initialized successfully {"addr": "rp-node-22-barely-engaging-crane:9092", "broker": "2"}\n12:29:15.531 DEBUG wrote Metadata v7 {"broker": "2", "bytes_written": 50, "write_wait": "783.486µs", "time_to_write": "9.177µs", "err": null}\n12:29:15.531 DEBUG read Metadata v7 {"broker": "2", "bytes_read": 271, "read_wait": "31.12µs", "time_to_read": "108.854µs", "err": null}\n12:29:15.531 DEBUG metadata refresh has identical topic partition data {"topic": "test-topic-post-upgrade-si", "partition": 0, "leader": 3, "leader_epoch": 1}\n12:29:15.532 DEBUG wrote Produce v7 {"broker": "3", "bytes_written": 131216, "write_wait": "78.374µs", "time_to_write": "42.482µs", "err": null}\n12:29:15.532 DEBUG read Produce v7 {"broker": "3", "bytes_read": 78, "read_wait": "211.015µs", "time_to_read": "141.561µs", "err": null}\n12:29:15.532 DEBUG retry batches processed {"wanted_metadata_update": true, "triggering_metadata_update": true, "should_backoff": false}\n12:29:15.532 DEBUG produced {"broker": "3", "to": "test-topic-post-upgrade-si[0{retrying@-1,1(UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.)}]"}\n12:29:15.532 INFO metadata update triggered {"why": "produce request had retry batches"}\n12:29:15.783 DEBUG wrote Metadata v7 {"broker": "2", "bytes_written": 50, "write_wait": "55.197µs", "time_to_write": "32.437µs", "err": null}\n12:29:15.783 DEBUG read Metadata v7 {"broker": "2", "bytes_read": 271, "read_wait": "82.135µs", "time_to_read": "200.897µs", "err": null}\n12:29:15.783 DEBUG metadata refresh has identical topic partition data {"topic": "test-topic-post-upgrade-si", "partition": 0, "leader": 3, "leader_epoch": 1}\n12:29:15.783 DEBUG wrote Produce v7 {"broker": "3", "bytes_written": 131216, "write_wait": "57.158µs", "time_to_write": "50.674µs", "err": null}\n12:29:15.784 DEBUG read Produce v7 {"broker": "3", "bytes_read": 78, "read_wait": "70.445µs", "time_to_read": "333.03µs", "err": null}\n12:29:15.784 DEBUG retry batches processed {"wanted_metadata_update": true, "triggering_metadata_update": true, "should_backoff": false}\n12:29:15.784 DEBUG produced {"broker": "3", "to": "test-topic-post-upgrade-si[0{retrying@-1,1(UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.)}]"}\n12:29:15.784 INFO metadata update triggered {"why": "produce request had retry batches"}\n12:29:16.034 DEBUG wrote Metadata v7 {"broker": "2", "bytes_written": 50, "write_wait": "50.853µs", "time_to_write": "34.991µs", "err": null}\n12:29:16.035 DEBUG read Metadata v7 {"broker": "2", "bytes_read": 271, "read_wait": "89.412µs", "time_to_read": "193.442µs", "err": null}\n12:29:16.035 DEBUG metadata refresh has identical topic partition data {"topic": "test-topic-post-upgrade-si", "partition": 0, "leader": 3, "leader_epoch": 1}\n12:29:16.035 DEBUG wrote Produce v7 {"broker": "3", "bytes_written": 131216, "write_wait": "44.499µs", "time_to_write": "46.899µs", "err": null}\n12:29:16.035 DEBUG read Produce v7 {"broker": "3", "bytes_read": 78, "read_wait": "36.232µs", "time_to_read": "418.887µs", "err": null}\n12:29:16.035 INFO batch in a produce request failed {"broker": "3", "topic": "test-topic-post-upgrade-si", "partition": 0, "err": "UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.", "err_is_retryable": true, "max_retries_reached": false}\n12:29:16.035 DEBUG produced {"broker": "3", "to": "test-topic-post-upgrade-si[0{err@-1,1(UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.)}]"}\nunable to produce record: UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.\n', 1, '')
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
data = self.run_test()
File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
return self.test_context.function(self.test)
File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 82, in wrapped
r = f(self, *args, **kwargs)
File "/home/ubuntu/redpanda/tests/rptest/tests/topic_creation_test.py", line 874, in test_retention_config_on_upgrade_from_v22_2_to_v22_3
self._populate_tiered_storage_topic(new_topic_name,
File "/home/ubuntu/redpanda/tests/rptest/tests/topic_creation_test.py", line 658, in _populate_tiered_storage_topic
self.rpk.produce(topic_name, "key", "b" * msg_size)
File "/home/ubuntu/redpanda/tests/rptest/clients/rpk.py", line 539, in produce
out = self._run_topic(cmd,
File "/home/ubuntu/redpanda/tests/rptest/clients/rpk.py", line 937, in _run_topic
return self._execute(cmd, stdin=stdin, timeout=timeout)
File "/home/ubuntu/redpanda/tests/rptest/clients/rpk.py", line 1067, in _execute
raise RpkException(
rptest.clients.rpk.RpkException: RpkException<command /opt/redpanda/bin/rpk topic -X brokers=rp-node-17-barely-engaging-crane:9092,rp-node-8-barely-engaging-crane:9092,rp-node-22-barely-engaging-crane:9092 produce --key key -z none --delivery-timeout 5s -f %v test-topic-post-upgrade-si -v returned 1, output: ; stderr: 12:29:15.224 INFO flushing
12:29:15.225 INFO producing to a new topic for the first time, fetching metadata to learn its partitions {"topic": "test-topic-post-upgrade-si"}
12:29:15.225 INFO immediate metadata update triggered {"why": "forced load because we are producing to a topic for the first time"}
12:29:15.225 DEBUG opening connection to broker {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "seed_0"}
12:29:15.226 DEBUG connection opened to broker {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "seed_0"}
12:29:15.226 DEBUG issuing api versions request {"broker": "seed_0", "version": 3}
12:29:15.226 DEBUG wrote ApiVersions v3 {"broker": "seed_0", "bytes_written": 31, "write_wait": "42.398µs", "time_to_write": "19.215µs", "err": null}
12:29:15.226 DEBUG read ApiVersions v3 {"broker": "seed_0", "bytes_read": 247, "read_wait": "37.133µs", "time_to_read": "292.958µs", "err": null}
12:29:15.226 DEBUG connection initialized successfully {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "seed_0"}
12:29:15.226 DEBUG wrote Metadata v7 {"broker": "seed_0", "bytes_written": 50, "write_wait": "1.68255ms", "time_to_write": "17.069µs", "err": null}
12:29:15.227 DEBUG read Metadata v7 {"broker": "seed_0", "bytes_read": 271, "read_wait": "35.189µs", "time_to_read": "221.844µs", "err": null}
12:29:15.227 INFO done waiting for metadata for new topic {"topic": "test-topic-post-upgrade-si"}
12:29:15.227 INFO initializing producer id
12:29:15.227 DEBUG opening connection to broker {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "3"}
12:29:15.227 DEBUG connection opened to broker {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "3"}
12:29:15.227 DEBUG issuing api versions request {"broker": "3", "version": 3}
12:29:15.227 DEBUG wrote ApiVersions v3 {"broker": "3", "bytes_written": 31, "write_wait": "6.763µs", "time_to_write": "14.027µs", "err": null}
12:29:15.228 DEBUG read ApiVersions v3 {"broker": "3", "bytes_read": 247, "read_wait": "18.149µs", "time_to_read": "327.305µs", "err": null}
12:29:15.228 DEBUG connection initialized successfully {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "3"}
12:29:15.228 DEBUG wrote InitProducerID v3 {"broker": "3", "bytes_written": 34, "write_wait": "871.293µs", "time_to_write": "8.515µs", "err": null}
12:29:15.229 DEBUG read InitProducerID v3 {"broker": "3", "bytes_read": 26, "read_wait": "30.908µs", "time_to_read": "1.477385ms", "err": null}
12:29:15.229 INFO producer id initialization success {"id": 1001, "epoch": 0}
12:29:15.229 DEBUG opening connection to broker {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "3"}
12:29:15.230 DEBUG connection opened to broker {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "3"}
12:29:15.230 DEBUG connection initialized successfully {"addr": "rp-node-17-barely-engaging-crane:9092", "broker": "3"}
12:29:15.230 DEBUG wrote Produce v7 {"broker": "3", "bytes_written": 131216, "write_wait": "467.908µs", "time_to_write": "109.659µs", "err": null}
12:29:15.230 DEBUG read Produce v7 {"broker": "3", "bytes_read": 78, "read_wait": "39.859µs", "time_to_read": "328.073µs", "err": null}
12:29:15.230 DEBUG retry batches processed {"wanted_metadata_update": true, "triggering_metadata_update": false, "should_backoff": true}
12:29:15.230 DEBUG produced {"broker": "3", "to": "test-topic-post-upgrade-si[0{retrying@-1,1(UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.)}]"}
12:29:15.530 DEBUG wrote Produce v7 {"broker": "3", "bytes_written": 131216, "write_wait": "62.625µs", "time_to_write": "63.885µs", "err": null}
12:29:15.530 DEBUG read Produce v7 {"broker": "3", "bytes_read": 78, "read_wait": "94.542µs", "time_to_read": "420.064µs", "err": null}
12:29:15.530 DEBUG retry batches processed {"wanted_metadata_update": true, "triggering_metadata_update": true, "should_backoff": false}
12:29:15.530 DEBUG produced {"broker": "3", "to": "test-topic-post-upgrade-si[0{retrying@-1,1(UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.)}]"}
12:29:15.530 INFO metadata update triggered {"why": "produce request had retry batches"}
12:29:15.530 DEBUG opening connection to broker {"addr": "rp-node-22-barely-engaging-crane:9092", "broker": "2"}
12:29:15.531 DEBUG connection opened to broker {"addr": "rp-node-22-barely-engaging-crane:9092", "broker": "2"}
12:29:15.531 DEBUG issuing api versions request {"broker": "2", "version": 3}
12:29:15.531 DEBUG wrote ApiVersions v3 {"broker": "2", "bytes_written": 31, "write_wait": "7.797µs", "time_to_write": "9.188µs", "err": null}
12:29:15.531 DEBUG read ApiVersions v3 {"broker": "2", "bytes_read": 247, "read_wait": "38.529µs", "time_to_read": "133.086µs", "err": null}
12:29:15.531 DEBUG connection initialized successfully {"addr": "rp-node-22-barely-engaging-crane:9092", "broker": "2"}
12:29:15.531 DEBUG wrote Metadata v7 {"broker": "2", "bytes_written": 50, "write_wait": "783.486µs", "time_to_write": "9.177µs", "err": null}
12:29:15.531 DEBUG read Metadata v7 {"broker": "2", "bytes_read": 271, "read_wait": "31.12µs", "time_to_read": "108.854µs", "err": null}
12:29:15.531 DEBUG metadata refresh has identical topic partition data {"topic": "test-topic-post-upgrade-si", "partition": 0, "leader": 3, "leader_epoch": 1}
12:29:15.532 DEBUG wrote Produce v7 {"broker": "3", "bytes_written": 131216, "write_wait": "78.374µs", "time_to_write": "42.482µs", "err": null}
12:29:15.532 DEBUG read Produce v7 {"broker": "3", "bytes_read": 78, "read_wait": "211.015µs", "time_to_read": "141.561µs", "err": null}
12:29:15.532 DEBUG retry batches processed {"wanted_metadata_update": true, "triggering_metadata_update": true, "should_backoff": false}
12:29:15.532 DEBUG produced {"broker": "3", "to": "test-topic-post-upgrade-si[0{retrying@-1,1(UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.)}]"}
12:29:15.532 INFO metadata update triggered {"why": "produce request had retry batches"}
12:29:15.783 DEBUG wrote Metadata v7 {"broker": "2", "bytes_written": 50, "write_wait": "55.197µs", "time_to_write": "32.437µs", "err": null}
12:29:15.783 DEBUG read Metadata v7 {"broker": "2", "bytes_read": 271, "read_wait": "82.135µs", "time_to_read": "200.897µs", "err": null}
12:29:15.783 DEBUG metadata refresh has identical topic partition data {"topic": "test-topic-post-upgrade-si", "partition": 0, "leader": 3, "leader_epoch": 1}
12:29:15.783 DEBUG wrote Produce v7 {"broker": "3", "bytes_written": 131216, "write_wait": "57.158µs", "time_to_write": "50.674µs", "err": null}
12:29:15.784 DEBUG read Produce v7 {"broker": "3", "bytes_read": 78, "read_wait": "70.445µs", "time_to_read": "333.03µs", "err": null}
12:29:15.784 DEBUG retry batches processed {"wanted_metadata_update": true, "triggering_metadata_update": true, "should_backoff": false}
12:29:15.784 DEBUG produced {"broker": "3", "to": "test-topic-post-upgrade-si[0{retrying@-1,1(UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.)}]"}
12:29:15.784 INFO metadata update triggered {"why": "produce request had retry batches"}
12:29:16.034 DEBUG wrote Metadata v7 {"broker": "2", "bytes_written": 50, "write_wait": "50.853µs", "time_to_write": "34.991µs", "err": null}
12:29:16.035 DEBUG read Metadata v7 {"broker": "2", "bytes_read": 271, "read_wait": "89.412µs", "time_to_read": "193.442µs", "err": null}
12:29:16.035 DEBUG metadata refresh has identical topic partition data {"topic": "test-topic-post-upgrade-si", "partition": 0, "leader": 3, "leader_epoch": 1}
12:29:16.035 DEBUG wrote Produce v7 {"broker": "3", "bytes_written": 131216, "write_wait": "44.499µs", "time_to_write": "46.899µs", "err": null}
12:29:16.035 DEBUG read Produce v7 {"broker": "3", "bytes_read": 78, "read_wait": "36.232µs", "time_to_read": "418.887µs", "err": null}
12:29:16.035 INFO batch in a produce request failed {"broker": "3", "topic": "test-topic-post-upgrade-si", "partition": 0, "err": "UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.", "err_is_retryable": true, "max_retries_reached": false}
12:29:16.035 DEBUG produced {"broker": "3", "to": "test-topic-post-upgrade-si[0{err@-1,1(UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.)}]"}
unable to produce record: UNKNOWN_TOPIC_OR_PARTITION: This server does not host this topic-partition.
; returncode: 1>
