CI Failure (Unexpected error seastar::nested_exception: std::__1::system_error (error system:32, sendmsg: Broken pipe)) in `RandomNodeOperationsTest.test_node_operations`

[vbotbuildovich]

https://buildkite.com/redpanda/vtools/builds/11696

Module: rptest.tests.random_node_operations_test
Class: RandomNodeOperationsTest
Method: test_node_operations
Arguments: {
    "num_to_upgrade": 0,
    "enable_failures": false,
    "with_tiered_storage": true
}

test_id:    RandomNodeOperationsTest.test_node_operations
status:     FAIL
run time:   1984.848 seconds

<BadLogLines nodes=ip-172-31-33-31(1) example="ERROR 2024-02-02 22:02:37,251 [shard 0:au  ] s3 - util.cc:105 - Unexpected error seastar::nested_exception: std::__1::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::__nested<std::__1::system_error> (error GnuTLS:-53, Error in the push function.): std::__1::system_error (error system:32, sendmsg: Broken pipe))">
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 168, in wrapped
    redpanda.raise_on_bad_logs(
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda.py", line 1413, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=ip-172-31-33-31(1) example="ERROR 2024-02-02 22:02:37,251 [shard 0:au  ] s3 - util.cc:105 - Unexpected error seastar::nested_exception: std::__1::system_error (error system:32, sendmsg: Broken pipe) (while cleaning up after std::__nested<std::__1::system_error> (error GnuTLS:-53, Error in the push function.): std::__1::system_error (error system:32, sendmsg: Broken pipe))">

JIRA Link: CORE-1754

[travisdowns]

In handle_client_transport_error where we categorize the error we already handle GnuTLS push error, we probably need to also handle the nested exception case as shown here. Have to check the seastar code but I guess while unwinding during the original GnuTLS exception we try to send more data down the pipe and get broken pipe (if this occurs or not in some run may be ~random depending on the exact state of the socket buffer).

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/12295

[piyushredpanda]

Not seen in at 6 weeks, closing