Windows Open SSH Server cannot support more than 512 concurrent ssh sessions (posix_spawn failing)

[golvellius1985]

trafficstars

Prerequisites

• [X] Write a descriptive title.
• [X] Make sure you are able to repro it on the latest version
• [X] Search the existing issues.

Steps to reproduce

Hi to all, I'm developing an application that needs to create a huge number of cuncurrent ssh sessions.

Server Operating System Windows 11 and Windows server 2016 (I have the same issue on both systems)

Client Operating System Windows 10 pro

Everything works well when I open 512 concurrent ssh sessions, but when I open the 513th concurrent session or more I receive on client side: connection reset

I see, if can help, that there is an old similar closed bug that had the same problem but with a smaller number of connections (50)

https://github.com/PowerShell/Win32-OpenSSH/issues/1096

Expected behavior

Support > 512 connections

Actual behavior

Stops every time at 512 exactly

Error details

Log on server side:

1452 2023-03-24 10:46:47.906 debug3: fd 6 is not O_NONBLOCK
1452 2023-03-24 10:46:47.906 debug3: spawning "C:\\Program Files\\OpenSSH\\sshd.exe" -R as subprocess
1452 2023-03-24 10:46:47.906 error: server_accept_loop, posix_spawn failed
1452 2023-03-24 10:46:47.906 debug3: send_rexec_state: entering fd = 10 config len 2205
1452 2023-03-24 10:46:47.906 debug3: ssh_msg_send: type 0
1452 2023-03-24 10:46:47.906 debug3: write ERROR from cb(2):232, io:0000014EA341F1D0
1452 2023-03-24 10:46:47.906 error: ssh_msg_send: write: Unknown error
1452 2023-03-24 10:46:47.906 error: send_rexec_state: ssh_msg_send failed
1452 2023-03-24 10:46:47.906 debug3: send_rexec_state: done
1452 2023-03-24 10:46:47.906 debug3: ReadFileEx() ERROR:109, io:0000014EA341F100

Environment data

Name                           Value
----                           -----
PSVersion                      5.1.14393.5582
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.14393.5582
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Version

OpenSSH_for_Windows_9.2p1, LibreSSL 3.6.1

Visuals

No response

[maertendMSFT]

Can you share more details on the application? Can the application be split so there is not the need for so many concurrent sessions? What would be a good upper limit in your mind?

[golvellius1985]

Hi, unfortunately my application can't be split, it is a springboot java application created with security purposes, it is mandatory for me to can use with a large number of concurrent ssh sessions. At this moment temporarily I can avoid the limit of 512 using a linux server but my final scope is to use windows server 2016.

For the question about a good upper limit, it could be possible to set this number in configuration file? for example inside sshd_config file? Otherwise if is not possible a perfect upper fixed limit would be 9000.

Thank a lot for your support.

[salvorizza]

A workaround could be increasing the MAX_CHILDREN and MAXIMUM_WAIT_OBJECTS_ENHANCED in contrib/win32/win32compat/signal_internal.h, i've already tried increasing this values and build, the process consumes a lot of RAM and works as expected, @maertendMSFT might be a good solution?

[matsmcp]

Since jumphosts and jumphost functionality seems to have become my thing.....

One connection through jumphosts can easily use four ssh sessions meaning a max of 128 real connections - still a lot but not impossible to reach.

The scenario in this case is an admin client that connects through an outgoing jumphost in its security zone (ssh session 1). From there the connection goes to the inbound jumphost in another security zone (ssh session 2). The connection now reaches the target server as a low privilege account - no ssh as admin over network (ssh session 3). Finally ssh administrator@localhost is used to elevate locally since we don't have sudo or runas under ssh (ssh session 4).

And yes you can do it as a single command ssh -J user@jumphost1,user@jumphost2,lowpriledgeuser@hostname administrator@localhost

[golvellius1985]

Hi, any news about this ticket? Thanks a lot