[Remote-SSH Bug]: too many connections - crash server

[david-torox]

Is there an existing issue for this bug?

• [X] I have searched the existing issues

Required Troubleshooting Steps

• [X] I have followed these troubleshooting steps
• [x] I have tried both values of the remote.SSH.useLocalServer setting

Connect Locally

It connects successfully

->

No response

Expected Behavior

when connecting to the remote server, we see around 30-60 processes spawn for each user using vs code, which eventually leads to the server crashing. i have followed several issues here about specific extensions causing the same issue but I use none of them(actually I don't use any extensions plus I disabled some of the builtin extensions some articles claim that are heavy on resources)

in the screenshots, you will see an example of the amount of processes that are spawned(seemingly for no reason)

Steps To Reproduce

1. open a vs code remote session

Remote-SSH Log

Remote-SSH Log

Log Level: 2
[12:32:42.329] SSH Resolver called for "ssh-remote+<ip>", attempt 1
[12:32:42.329] "remote.SSH.useLocalServer": false
[12:32:42.330] "remote.SSH.useExecServer": false
[12:32:42.330] "remote.SSH.showLoginTerminal": false
[12:32:42.330] "remote.SSH.remotePlatform": {"<ip>":"linux"}
[12:32:42.330] "remote.SSH.path": undefined
[12:32:42.330] "remote.SSH.configFile": undefined
[12:32:42.330] "remote.SSH.useFlock": true
[12:32:42.330] "remote.SSH.lockfilesInTmp": false
[12:32:42.330] "remote.SSH.localServerDownload": auto
[12:32:42.330] "remote.SSH.remoteServerListenOnSocket": false
[12:32:42.330] "remote.SSH.showLoginTerminal": false
[12:32:42.330] "remote.SSH.defaultExtensions": []
[12:32:42.330] "remote.SSH.loglevel": 2
[12:32:42.330] "remote.SSH.enableDynamicForwarding": true
[12:32:42.330] "remote.SSH.enableRemoteCommand": false
[12:32:42.330] "remote.SSH.serverPickPortsFromRange": {}
[12:32:42.330] "remote.SSH.serverInstallPath": {}
[12:32:42.333] VS Code version: 1.85.2
[12:32:42.333] Remote-SSH version: remote-ssh@
[12:32:42.333] win32 x64
[12:32:42.339] SSH Resolver called for host: <ip>
[12:32:42.339] Setting up SSH remote "<ip>"
[12:32:42.341] Using commit id "id" and quality "stable" for server
[12:32:42.342] Install and start server if needed
[12:32:42.343] Checking ssh with "C:\Windows\system32\ssh.exe -V"
[12:32:42.346] Got error from ssh: spawn C:\Windows\system32\ssh.exe ENOENT
[12:32:42.346] Checking ssh with "C:\Windows\ssh.exe -V"
[12:32:42.347] Got error from ssh: spawn C:\Windows\ssh.exe ENOENT
[12:32:42.347] Checking ssh with "C:\Windows\System32\Wbem\ssh.exe -V"
[12:32:42.348] Got error from ssh: spawn C:\Windows\System32\Wbem\ssh.exe ENOENT
[12:32:42.348] Checking ssh with "C:\Windows\System32\WindowsPowerShell\v1.0\ssh.exe -V"
[12:32:42.350] Got error from ssh: spawn C:\Windows\System32\WindowsPowerShell\v1.0\ssh.exe ENOENT
[12:32:42.350] Checking ssh with "C:\Windows\System32\OpenSSH\ssh.exe -V"
[12:32:42.372] > OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3

[12:32:42.375] Running script with connection command: "C:\Windows\System32\OpenSSH\ssh.exe" -T -D port "<ip>" bash
[12:32:42.376] Terminal shell path: C:\Windows\System32\cmd.exe
[12:32:45.261] > : running
> ]0;C:\Windows\System32\cmd.exe Acquiring lock on /home/david/.vscode-server/bin//vscode-remote-lock.david.
> Found existing installation at /home/david/.vscode-server/bin/...
> Checking /home/david/.vscode-server/..log and /home/david/.vscode-server/..pid for a running server
> Looking for server with pid: <pid>
[12:32:45.261] Got some output, clearing connection timeout
[12:32:45.268] > Found running server...
>  
> *
> * Visual Studio Code Server
> *
> * By using the software, you agree to
[12:32:45.370] > * the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-license) and
> * the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacystatement).
> *
>  
> Checking server status on port <port> with wget
> : start
> SSH_AUTH_SOCK====
> DISPLAY====
> webUiAccessToken====
> listeningOn====
> osReleaseId==ubuntu==
> arch==x86_64==
> vscodeArch==x64==
> bitness==64==
> tmpDir==/run/user/1003==
> platform==linux==
> unpackResult====
> didLocalDownload==0==
> downloadTime====
> installTime====
> extInstallTime====
> serverStartTime====
> connectionToken==<ct>==
> c23c4e9c5930: end
[12:32:45.370] Received install output: 
SSH_AUTH_SOCK====
DISPLAY====
webUiAccessToken====
listeningOn==<port>==
osReleaseId==ubuntu==
arch==x86_64==
vscodeArch==x64==
bitness==64==
tmpDir==/run/user/1003==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====
extInstallTime====
serverStartTime====
connectionToken==<ct>==

[12:32:45.370] Remote server is listening on port <port>
[12:32:45.370] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":},"osReleaseId":"ubuntu","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/1003","platform":"linux","connectionToken":""},"installUnpackCode":""}
[12:32:45.372] Starting forwarding server. local port  -> socksPort  -> remotePort 
[12:32:45.372] Forwarding server listening on port 
[12:32:45.372] Waiting for ssh tunnel to be ready
[12:32:45.373] [Forwarding server port ] Got connection 0
[12:32:45.374] Tunneled port  to local port 
[12:32:45.374] Resolved "ssh-remote+" to "port "
[12:32:45.378] ------




[12:32:45.393] > 
[12:32:45.393] [Forwarding server port 52270] Got connection 1
[12:32:45.713] [Forwarding server port 52270] Got connection 2

Anything else?

I have no extensions installed except for the remote SSH one. I have uninstalled several built-in extensions.
this is happening for all of our vscode users who are connecting with a remote-ssh extension

all the processes are new and not leftovers from previous connections, all connections close within 2-3 minutes after disconnection from the remote

[roblourens]

These are not all processes, you are looking at threads in htop. But there are several processes that the VS Code server uses and a certain amount of RAM is needed. Is it possible that your remote host doesn't have enough RAM to support it?

[david-torox]

An update from another incident, according to our DevOps guy the server does not actually crash, it's just that connecting to it is getting blocked for SSH or for specific users(not sure yet)

We know that because we have an agent on the instance that is supposed to notify us when the connection to the server is lost, and we get no such notification.

As for the server specs, it has 4GB of RAM, around the area of the crash there is no dangerously high RAM usage, the same goes for the CPU, disk writes and packets received/sent. I'll add some screenshots from our DevOps guy.

mem usage:

network usage:

CPU usage:

[david-torox]

is there any update @roblourens?

[vscodenpa]

Hey @roblourens, this issue might need further attention.

@david-torox, you can help us out by closing this issue if the problem no longer exists, or adding more information.

[vscodenpa]

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!