redpanda icon indicating copy to clipboard operation
redpanda copied to clipboard

CI Failure (key symptom) in `UpgradeBackToBackTest.test_upgrade_with_all_workloads`

Open vbotbuildovich opened this issue 9 months ago • 0 comments

https://buildkite.com/redpanda/vtools/builds/13742

Module: rptest.tests.upgrade_test
Class: UpgradeBackToBackTest
Method: test_upgrade_with_all_workloads
Arguments: {
    "single_upgrade": false
}
test_id:    UpgradeBackToBackTest.test_upgrade_with_all_workloads
status:     FAIL
run time:   131.186 seconds

RemoteCommandError({'ssh_config': {'host': 'ip-172-31-3-81', 'hostname': '172.31.3.81', 'user': 'root', 'port': 22, 'password': None, 'identityfile': '/home/ubuntu/.ssh/id_rsa'}, 'hostname': 'ip-172-31-3-81', 'ssh_hostname': '172.31.3.81', 'user': 'root', 'externally_routable_ip': '35.161.143.84', '_logger': <Logger rptest.tests.upgrade_test.UpgradeBackToBackTest.test_upgrade_with_all_workloads.single_upgrade=False-822 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7f3ce17ec0d0>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7f3ce164b7f0>, '_custom_ssh_exception_checks': None}, 'curl -fsSL https://vectorized-public.s3.us-west-2.amazonaws.com/releases/redpanda/22.2.13/redpanda-22.2.13-amd64.tar.gz --retry 3 --retry-connrefused --retry-delay 2 --create-dir -o /opt/redpanda_installs/v22.2.13/redpanda.tar.gz && gunzip -c /opt/redpanda_installs/v22.2.13/redpanda.tar.gz | tar -xf - -C /opt/redpanda_installs/v22.2.13 && rm /opt/redpanda_installs/v22.2.13/redpanda.tar.gz', 35, b'')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 103, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/upgrade_test.py", line 224, in test_upgrade_with_all_workloads
    for current_version in self.upgrade_through_versions(self.versions):
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_test.py", line 247, in upgrade_through_versions
    current_version = install_next()
  File "/home/ubuntu/redpanda/tests/rptest/tests/redpanda_test.py", line 174, in install_next
    self.redpanda._installer.install(self.redpanda.nodes, v)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 609, in install
    self._install_unlocked(nodes, install_target)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 658, in _install_unlocked
    raise e
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 638, in _install_unlocked
    self.wait_for_async_ssh(self._redpanda.logger,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 165, in wait_for_async_ssh
    for l in ssh_out_per_node[node]:
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 687, in next
    return next(self.iter_obj)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 354, in output_generator
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-3-81: Command 'curl -fsSL https://vectorized-public.s3.us-west-2.amazonaws.com/releases/redpanda/22.2.13/redpanda-22.2.13-amd64.tar.gz --retry 3 --retry-connrefused --retry-delay 2 --create-dir -o /opt/redpanda_installs/v22.2.13/redpanda.tar.gz && gunzip -c /opt/redpanda_installs/v22.2.13/redpanda.tar.gz | tar -xf - -C /opt/redpanda_installs/v22.2.13 && rm /opt/redpanda_installs/v22.2.13/redpanda.tar.gz' returned non-zero exit status 35.

JIRA Link: CORE-2976

vbotbuildovich avatar May 15 '24 21:05 vbotbuildovich