CI Failure (paramiko.buffered_pipe.PipeTimeout - produce timeout) in `RecoveryModeTest.test_recovery_mode`

[bharathv]

https://buildkite.com/redpanda/redpanda/builds/42222

Module: rptest.tests.recovery_mode_test
Class: RecoveryModeTest
Method: test_recovery_mode

test_id:    RecoveryModeTest.test_recovery_mode
status:     FAIL
run time:   78.157 seconds

Exception('RpkProducer-1-140565560335600-worker-1: Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/dist-packages/paramiko/channel.py", line 699, in recv\n    out = self.in_buffer.read(nbytes, self.timeout)\n  File "/usr/local/lib/python3.10/dist-packages/paramiko/buffered_pipe.py", line 164, in read\n    raise PipeTimeout()\nparamiko.buffered_pipe.PipeTimeout\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 38, in _protected_worker\n    self._worker(idx, node)\n  File "/root/tests/rptest/services/rpk_producer.py", line 73, in _worker\n    for line in node.account.ssh_capture(\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 687, in next\n    return next(self.iter_obj)\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 344, in output_generator\n    for line in iter(stdout.readline, \'\'):\n  File "/usr/local/lib/python3.10/dist-packages/paramiko/file.py", line 291, in readline\n    new_data = self._read(n)\n  File "/usr/local/lib/python3.10/dist-packages/paramiko/channel.py", line 1361, in _read\n    return self.channel.recv(size)\n  File "/usr/local/lib/python3.10/dist-packages/paramiko/channel.py", line 701, in recv\n    raise socket.timeout()\nTimeoutError\n')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 82, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/recovery_mode_test.py", line 179, in test_recovery_mode
    _produce(self, "mytopic1")
  File "/root/tests/rptest/tests/recovery_mode_test.py", line 33, in _produce
    producer.run()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/service.py", line 345, in run 
    self.wait()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 74, in wait
    self._propagate_exceptions()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 100, in _propagate_exceptions
    raise Exception(self.errors)
Exception: RpkProducer-1-140565560335600-worker-1: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/paramiko/channel.py", line 699, in recv
    out = self.in_buffer.read(nbytes, self.timeout)
  File "/usr/local/lib/python3.10/dist-packages/paramiko/buffered_pipe.py", line 164, in read
    raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 38, in _protected_worker
    self._worker(idx, node)
  File "/root/tests/rptest/services/rpk_producer.py", line 73, in _worker
    for line in node.account.ssh_capture(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 687, in next
    return next(self.iter_obj)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/cluster/remoteaccount.py", line 344, in output_generator
    for line in iter(stdout.readline, ''):
  File "/usr/local/lib/python3.10/dist-packages/paramiko/file.py", line 291, in readline
    new_data = self._read(n)
  File "/usr/local/lib/python3.10/dist-packages/paramiko/channel.py", line 1361, in _read
    return self.channel.recv(size)
  File "/usr/local/lib/python3.10/dist-packages/paramiko/channel.py", line 701, in recv
    raise socket.timeout()
TimeoutError

[bharathv]

cc: @ztlpn

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/42465

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/43625

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/44197

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/11824

[dotnwat]

Moved this to devprod team as its more about ducktape infrastructure.

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/45479

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/45560

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/46682

[piyushredpanda]

Not seen in at least two months, closing