failure in PartitionMovementUpgradeTest.test_basic_upgrade

[dotnwat]

https://buildkite.com/redpanda/redpanda/builds/13571#0182669b-0537-49b4-9b5a-bb45b6114e90/6-9016

Exception('FranzGoVerifiableConsumerGroupConsumer-0-140208812418784-worker-1: Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/root/tests/rptest/services/franz_go_verifiable_services.py", line 284, in _worker\n    self.save_exception(ex)\n  File "/root/tests/rptest/services/franz_go_verifiable_services.py", line 84, in save_exception\n    raise ex\n  File "/root/tests/rptest/services/franz_go_verifiable_services.py", line 267, in _worker\n    f"--brokers={self._redpanda.brokers()} "\n  File "/root/tests/rptest/services/redpanda.py", line 1606, in brokers\n    return ",".join(self.brokers_list(limit))\n  File "/root/tests/rptest/services/redpanda.py", line 1609, in brokers_list\n    brokers = [self.broker_address(n) for n in self._started[:limit]]\n  File "/root/tests/rptest/services/redpanda.py", line 1609, in <listcomp>\n    brokers = [self.broker_address(n) for n in self._started[:limit]]\n  File "/root/tests/rptest/services/redpanda.py", line 1589, in broker_address\n    assert node in self._started\nAssertionError\n')
  | Traceback (most recent call last):
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
  | data = self.run_test()
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
  | return self.test_context.function(self.test)
  | File "/root/tests/rptest/services/cluster.py", line 35, in wrapped
  | r = f(self, *args, **kwargs)
  | File "/root/tests/rptest/tests/partition_movement_upgrade_test.py", line 140, in test_basic_upgrade
  | self.verify()
  | File "/root/tests/rptest/tests/partition_movement_upgrade_test.py", line 71, in verify
  | self.consumer.wait()
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 74, in wait
  | self._propagate_exceptions()
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 100, in _propagate_exceptions
  | raise Exception(self.errors)
  | Exception: FranzGoVerifiableConsumerGroupConsumer-0-140208812418784-worker-1: Traceback (most recent call last):
  | File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 38, in _protected_worker
  | self._worker(idx, node)
  | File "/root/tests/rptest/services/franz_go_verifiable_services.py", line 284, in _worker
  | self.save_exception(ex)
  | File "/root/tests/rptest/services/franz_go_verifiable_services.py", line 84, in save_exception
  | raise ex
  | File "/root/tests/rptest/services/franz_go_verifiable_services.py", line 267, in _worker
  | f"--brokers={self._redpanda.brokers()} "
  | File "/root/tests/rptest/services/redpanda.py", line 1606, in brokers
  | return ",".join(self.brokers_list(limit))
  | File "/root/tests/rptest/services/redpanda.py", line 1609, in brokers_list
  | brokers = [self.broker_address(n) for n in self._started[:limit]]
  | File "/root/tests/rptest/services/redpanda.py", line 1609, in <listcomp>
  | brokers = [self.broker_address(n) for n in self._started[:limit]]
  | File "/root/tests/rptest/services/redpanda.py", line 1589, in broker_address
  | assert node in self._started
  | AssertionError
 

<br class="Apple-interchange-newline">

[andrwng]

Looks like the test starts up a FranzGoVerifiableProducer, which in the background calls redpanda.brokers() to put together the appropriate command. In the foreground, the test restarts nodes, and as a part of restarting, nodes are removed and added to the _started nodes.

We should probably call redpanda.brokers() and cache the value in the producer before running the workload.