CI Failure (`tsh ssh` command failed) in `OMBValidationTest.test_max_partitions`

[vbotbuildovich]

https://buildkite.com/redpanda/vtools/builds/12674

Module: rptest.redpanda_cloud_tests.omb_validation_test
Class: OMBValidationTest
Method: test_max_partitions

test_id:    OMBValidationTest.test_max_partitions
status:     FAIL
run time:   769.277 seconds

CalledProcessError(1, ['tsh', 'ssh', '--proxy=proxy.tp.redpanda.com:443', '--auth=okta', '--identity=/tmp/machine-id/identity', 'redpanda@co70pk3mkj6ngclnifi0-agent', 'kubectl', '--context=gke_devprod-cicd-infra_us-west1_redpanda-co70pk3mkj6ngclnifi0', 'run', '--image docker.io/library/alpine', '--restart=Never', '--overrides=\'{"spec": {"nodeName": "gke-redpanda-co70pk3mkj-redpanda-8e5f-4eb9cdcf-0zz9", "hostPID": true, "hostNetwork": true, "containers": [{"securityContext": {"privileged": true}, "image": "docker.io/library/alpine", "name": "nsenter", "stdin": true, "stdinOnce": true, "tty": true, "command": ["nsenter", "--target", "1", "--mount", "--uts", "--ipc", "--net", "--pid", "bash", "-l"], "resources": {}, "volumeMounts": []}], "tolerations": [{"key": "CriticalAddonsOnly", "operator": "Exists"}, {"effect": "NoExecute", "operator": "Exists"}], "volumes": []}}\'', 'gke-redpanda-co70pk3mkj-redpanda-8e5f-4eb9cdcf-0zz9-priviledged-shell'])
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 177, in wrapped
    redpanda.raise_on_bad_logs(
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 2297, in raise_on_bad_logs
    lsearcher.search_logs(self.get_redpanda_pods())
  File "/home/ubuntu/redpanda/tests/rptest/services/utils.py", line 153, in search_logs
    bad_loglines = self._search(nodes)
  File "/home/ubuntu/redpanda/tests/rptest/services/utils.py", line 134, in _search
    for line in self._capture_log(node, self.match_expr):
  File "/home/ubuntu/redpanda/tests/rptest/services/utils.py", line 186, in _capture_log
    with KubeNodeShell(self.kubectl, node_name) as ksh:
  File "/home/ubuntu/redpanda/tests/rptest/clients/kubectl.py", line 383, in __enter__
    _out = self.kubectl.cmd([
  File "/home/ubuntu/redpanda/tests/rptest/clients/kubectl.py", line 179, in cmd
    return self._cmd(cmd)
  File "/home/ubuntu/redpanda/tests/rptest/clients/kubectl.py", line 160, in _cmd
    return subprocess.check_output(remote_cmd, stderr=subprocess.PIPE)
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['tsh', 'ssh', '--proxy=proxy.tp.redpanda.com:443', '--auth=okta', '--identity=/tmp/machine-id/identity', 'redpanda@co70pk3mkj6ngclnifi0-agent', 'kubectl', '--context=gke_devprod-cicd-infra_us-west1_redpanda-co70pk3mkj6ngclnifi0', 'run', '--image docker.io/library/alpine', '--restart=Never', '--overrides=\'{"spec": {"nodeName": "gke-redpanda-co70pk3mkj-redpanda-8e5f-4eb9cdcf-0zz9", "hostPID": true, "hostNetwork": true, "containers": [{"securityContext": {"privileged": true}, "image": "docker.io/library/alpine", "name": "nsenter", "stdin": true, "stdinOnce": true, "tty": true, "command": ["nsenter", "--target", "1", "--mount", "--uts", "--ipc", "--net", "--pid", "bash", "-l"], "resources": {}, "volumeMounts": []}], "tolerations": [{"key": "CriticalAddonsOnly", "operator": "Exists"}, {"effect": "NoExecute", "operator": "Exists"}], "volumes": []}}\'', 'gke-redpanda-co70pk3mkj-redpanda-8e5f-4eb9cdcf-0zz9-priviledged-shell']' returned non-zero exit status 1.

JIRA Link: CORE-2090

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12758 *https://buildkite.com/redpanda/vtools/builds/12771 *https://buildkite.com/redpanda/vtools/builds/12789 *https://buildkite.com/redpanda/vtools/builds/12788

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/13187

[rpdevmp]

Duplicate of https://github.com/redpanda-data/redpanda/issues/18513