Potential issue with handling multiple channels/premature EOF

[mihirsamdarshi]

I've had an issue I've been trying to debug this for a while now and hopefully try to upstream whatever fix it takes, but I think I am at my wit's end trying to debug at this point.

I have been using Russh to try to create a local port-forwarding utility that will end up being embedded into my application. However I am running into an issue. For this, I use the client module's session.channel_open_direct_tcpip method. For reference and reproduction, the utility (that works via CLI) can be found here.

The main issue is that when sending a large amount of requests via an ClientSession (and therefore via multiple open channels) there is a massive delay between sending the request and getting any sort of response. (this is not a server issue, it works perfectly fine when running over OpenSSH).

You won't see this behavior being exhibited (in my utility) when you send a single curl request over the session (therefore opening and closing one channel), but it will be noticeable if you try to open a webpage over the connection that sends >5 or 6 XHR requests over the connection.

I have narrowed down the source of this slow down to be somewhat related to the select! macro in the Session run function, and suspect this has to do with many channels being opened but not being closed when a full TCP stream is sent back.

I figured out that OpenSSH actually just closes the channel when it cannot read from the input or output buffer any more, so I thought the fix for this would have been sending a channel EOF immediately after forwarding the local TCP stream to the remote (at around line 148 in the utility), but it seems the OpenSSH (or Russh?) takes this as a sign I want to close the channel and just drops it immediately.

Any help on even where to dig deeper would be much appreciated

[Eugeny]

I wasn't able to find the root cause yet, but sending an EOF after receiving the first response byte seems to work (this workaround obviously only makes sense for request-respond protocols like HTTP): https://github.com/Eugeny/ssh-test/commit/600b51d831445d4dc1ba8bb230798168a8b35461

[mihirsamdarshi]

Thanks so much @Eugeny I am going to continue working on debugging this. I am currently deep into concurrent lldb sessions comparing OpenSSH and Russh lol. I'll continue pushing whatever changes I make to both my testing repo and my fork of Russh.