redpanda CI Failure (key symptom) in `RedpandaUpgradeTest.test_workloads_through

https://buildkite.com/redpanda/vtools/builds/13607

Module: rptest.tests.workload_upgrade_runner_test
Class: RedpandaUpgradeTest
Method: test_workloads_through_releases
Arguments: {
    "cloud_storage_type": 1
}

test_id:    RedpandaUpgradeTest.test_workloads_through_releases
status:     FAIL
run time:   1050.143 seconds

TimeoutError('Redpanda service ip-172-31-7-94 failed to start within 90 sec')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 103, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/workload_upgrade_runner_test.py", line 278, in test_workloads_through_releases
    for current_version in self.upgrade_through_versions(
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_test.py", line 259, in upgrade_through_versions
    self.redpanda.rolling_restart_nodes(
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1365, in rolling_restart_nodes
    restarter.restart_nodes(nodes,
  File "/home/ubuntu/redpanda/tests/rptest/services/rolling_restarter.py", line 100, in restart_nodes
    self.redpanda.start_node(
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2871, in start_node
    self.start_service(node, start_rp)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2947, in start_service
    start()
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2862, in start_rp
    wait_until(
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Redpanda service ip-172-31-7-94 failed to start within 90 sec

JIRA Link: CORE-2877

May 09 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/13628

May 10 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/13649

May 11 '24 21:05 vbotbuildovich

*https://buildkite.com/redpanda/vtools/builds/13663

May 12 '24 21:05 vbotbuildovich