[BUG] RemoteClusterClientTests.testConnectAndExecuteRequest is flaky

[peternied]

Describe the bug

org.opensearch.transport.RemoteClusterClientTests.testConnectAndExecuteRequest seems to be able to get network exceptions during this workflow

2> junit.framework.AssertionFailedError: Unexpected exception type, expected ActionNotFoundTransportException but got NodeDisconnectedException[[remote_node][127.0.0.1:10600][indices:data/read/scroll] disconnected]
      at org.apache.lucene.tests.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2894)
      at org.apache.lucene.tests.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2875)
      at org.opensearch.transport.RemoteClusterClientTests.testConnectAndExecuteRequest(RemoteClusterClientTests.java:103)

      Caused by:
      NodeDisconnectedException[[remote_node][127.0.0.1:10600][indices:data/read/scroll] disconnected]

  java.lang.AssertionError
      at __randomizedtesting.SeedInfo.seed([C42E5842F94566E0]:0)
      at org.opensearch.transport.InboundMessage.openOrGetStreamInput(InboundMessage.java:116)
      at org.opensearch.transport.TransportLogger.format(TransportLogger.java:150)
      at org.opensearch.transport.TransportLogger.logInboundMessage(TransportLogger.java:70)
      at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:123)
      at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:770)
      at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175)
      at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150)
      at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115)
      at org.opensearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:343)
      at org.opensearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:246)
      at org.opensearch.nio.BytesChannelContext.read(BytesChannelContext.java:59)
      at org.opensearch.nio.EventHandler.handleRead(EventHandler.java:152)
      at org.opensearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:167)
      at org.opensearch.nio.NioSelector.handleRead(NioSelector.java:438)
      at org.opensearch.nio.NioSelector.processKey(NioSelector.java:264)
      at org.opensearch.nio.NioSelector.singleLoop(NioSelector.java:191)
      at org.opensearch.nio.NioSelector.runLoop(NioSelector.java:148)
      at java.base/java.lang.Thread.run(Thread.java:1583)

Related component

Storage:Remote

To Reproduce

Initial failure on developer desktop, was not able to reproduce it on rerun

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.transport.RemoteClusterClientTests.testConnectAndExecuteRequest" -Dtests.seed=C42E5842F94566E0 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=nl-NL -Dtests.timezone=Pacific/Wallis -Druntime.java=21

Expected behavior

All tests should pass reliably

Additional Details

Host/Environment (please complete the following information):

% uname -a
Linux dev-dsk-petern 5.10.209-175.812.amzn2int.x86_64 #1 SMP Tue Jan 30 21:29:45 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux