Remote-ssh stuck

[Priya22]

Hi, I usually connect to a remote server using the VS Code Remote-ssh extension. Since yesterday, I'm unable to establish a connection to the host. The attempt fails with the following log trace:

[15:44:27.964] Log Level: 1
[15:44:27.968] [email protected]
[15:44:27.968] linux x64
[15:44:27.975] SSH Resolver called for "ssh-remote+vremote.vectorinstitute.ai", attempt 1
[15:44:27.975] "remote.SSH.useLocalServer": false
[15:44:27.976] "remote.SSH.showLoginTerminal": true
[15:44:27.976] "remote.SSH.remotePlatform": {}
[15:44:27.976] "remote.SSH.path": undefined
[15:44:27.976] "remote.SSH.configFile": undefined
[15:44:27.977] "remote.SSH.useFlock": true
[15:44:27.977] "remote.SSH.lockfilesInTmp": false
[15:44:27.977] "remote.SSH.localServerDownload": auto
[15:44:27.977] "remote.SSH.remoteServerListenOnSocket": true
[15:44:27.978] "remote.SSH.showLoginTerminal": true
[15:44:27.978] "remote.SSH.defaultExtensions": []
[15:44:27.978] "remote.SSH.loglevel": 1
[15:44:27.979] "remote.SSH.enableDynamicForwarding": true
[15:44:27.979] "remote.SSH.enableRemoteCommand": false
[15:44:27.979] "remote.SSH.serverPickPortsFromRange": {}
[15:44:27.979] "remote.SSH.serverInstallPath": {}
[15:44:27.984] SSH Resolver called for host: vremote.vectorinstitute.ai
[15:44:27.984] Setting up SSH remote "vremote.vectorinstitute.ai"
[15:44:27.991] Using commit id "e7f30e38c5a4efafeec8ad52861eb772a9ee4dfb" and quality "stable" for server
[15:44:27.997] Install and start server if needed
[15:44:30.175] PATH: /home/krishnapriya/bin:/home/krishnapriya/.local/bin:/snap/bin:/home/krishnapriya/gems/bin:/home/krishnapriya/anaconda3/bin:/home/krishnapriya/anaconda3/condabin:/home/krishnapriya/bin:/home/krishnapriya/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/krishnapriya/.local/share/JetBrains/Toolbox/scripts:/home/krishnapriya/.local/share/JetBrains/Toolbox/scripts
[15:44:30.177] Checking ssh with "ssh -V"
[15:44:30.230] > OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020

[15:44:30.245] Running script with connection command: ssh -T -D 41635 -o ConnectTimeout=15 'vremote.vectorinstitute.ai' bash
[15:44:31.236] "install" terminal received data: "
Duo two-factor login for vkpriya

Enter a passcode or select one of the following options:

 1. Duo Push to XXX-XXX-1465
 2. SMS passcodes to XXX-XXX-1465

Passcode or option (1-2): "
[15:44:31.236] Got some output, clearing connection timeout
[15:44:31.793] "install" terminal received data: "1"
[15:44:32.216] "install" terminal received data: ""
[15:44:38.408] "install" terminal received data: "37ea395c9987: running"
[15:44:38.473] "install" terminal received data: "Acquiring lock on /h/vkpriya/.vscode-server/bin/e7f30e38c5a4efafeec8ad52861eb772a9ee4dfb/vscode-remote-lock.vkpriya.e7f30e38c5a4efafeec8ad52861eb772a9ee4dfb"

I've tried a bunch of suggestions from previous issues:

1. Removing/renaming the .vscode and .vscode-server folders on the remote host
2. Toggling the remote-ssh useLocalServer, useFlock settings
3. Waiting

None of these work. Any suggestions, or ideas on what might be going wrong?

Thanks!

[doseonfairsutdios]

same issues. It seems to be related to the vscode site speed. Windows vscode portable file downloads are incredibly slow and even fail many times. I hope the issue will be fixed as soon as possible.

[leeminju531]

I experienced the same issue #7188 I think too it's occured due to the vscode site speed.

[eleanorjboyd]

Hello! Could those who are experiencing this issue please send over the remote-ssh console logs? In regards to the initial issue is the line [15:44:38.473] "install" terminal received data: "Acquiring lock on /h/vkpriya/..... the final log or is there additional information? Trying to find the exact error message to see if that would give insight. Thanks!

[vinicius507]

Hello! Could those who are experiencing this issue please send over the remote-ssh console logs? In regards to the initial issue is the line [15:44:38.473] "install" terminal received data: "Acquiring lock on /h/vkpriya/..... the final log or is there additional information? Trying to find the exact error message to see if that would give insight. Thanks!

I'm having this exact same problem. The log message Acquiring lock on [...] is indeed the last message that appears.

Here are my logs:

[15:40:53.932] Log Level: 2
[15:40:53.932] [email protected]
[15:40:53.933] linux x64
[15:40:53.936] SSH Resolver called for "ssh-remote+<REDACTED>", attempt 1
[15:40:53.936] "remote.SSH.useLocalServer": true
[15:40:53.937] "remote.SSH.path": undefined
[15:40:53.937] "remote.SSH.configFile": undefined
[15:40:53.937] "remote.SSH.useFlock": true
[15:40:53.937] "remote.SSH.lockfilesInTmp": false
[15:40:53.937] "remote.SSH.localServerDownload": auto
[15:40:53.937] "remote.SSH.remoteServerListenOnSocket": false
[15:40:53.937] "remote.SSH.showLoginTerminal": false
[15:40:53.937] "remote.SSH.defaultExtensions": []
[15:40:53.938] "remote.SSH.loglevel": 2
[15:40:53.938] "remote.SSH.enableDynamicForwarding": true
[15:40:53.938] "remote.SSH.enableRemoteCommand": false
[15:40:53.938] "remote.SSH.serverPickPortsFromRange": {}
[15:40:53.938] "remote.SSH.serverInstallPath": {}
[15:40:53.944] SSH Resolver called for host: <REDACTED>
[15:40:53.944] Setting up SSH remote "<REDACTED>"
[15:40:53.947] Acquiring local install lock: /tmp/vscode-remote-ssh-6921d55c-install.lock
[15:40:53.950] Looking for existing server data file at /home/vinicius/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-6921d55c-74b1f979648cc44d385a2286793c226e611f59e7-0.84.0/data.json
[15:40:53.951] Using commit id "74b1f979648cc44d385a2286793c226e611f59e7" and quality "stable" for server
[15:40:53.954] Install and start server if needed
[15:40:53.967] PATH: /home/vinicius/.cargo/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/home/vinicius/.dotnet/tools:/var/lib/flatpak/exports/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
[15:40:53.967] Checking ssh with "ssh -V"
[15:40:53.972] > OpenSSH_9.0p1, OpenSSL 1.1.1q  5 Jul 2022

[15:40:53.974] askpass server listening on /run/user/1000/vscode-ssh-askpass-d184807af8e8cab3e6238b3945f3ca30767b9ea2.sock
[15:40:53.974] Spawning local server with {"serverId":1,"ipcHandlePath":"/run/user/1000/vscode-ssh-askpass-7103acc83019316d0977998c95ffca43a63360e1.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","42401","-o","ConnectTimeout=15","<REDACTED>"],"serverDataFolderName":".vscode-server","dataFilePath":"/home/vinicius/.config/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-6921d55c-74b1f979648cc44d385a2286793c226e611f59e7-0.84.0/data.json"}
[15:40:53.975] Local server env: {"SSH_AUTH_SOCK":"/run/user/1000/keyring/ssh","SHELL":"/usr/bin/zsh","DISPLAY":":0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/home/vinicius/.vscode/extensions/ms-vscode-remote.remote-ssh-0.84.0/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/opt/visual-studio-code/code","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"/home/vinicius/.vscode/extensions/ms-vscode-remote.remote-ssh-0.84.0/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/run/user/1000/vscode-ssh-askpass-d184807af8e8cab3e6238b3945f3ca30767b9ea2.sock"}
[15:40:53.988] Spawned 23829
[15:40:54.065] > local-server-1> Spawned ssh, pid=23837
[15:40:54.066] stderr> OpenSSH_9.0p1, OpenSSL 1.1.1q  5 Jul 2022
[15:40:54.535] stderr> debug1: Server host key: ssh-ed25519 SHA256:cuMAeo4xwC0/Q6OOhIxsUmEvJsKFn9Eu1hYCOoSq5g8
[15:40:54.851] stderr> Welcome to 42 OpenSSH Server workspace
[15:40:54.920] Got askpass request: {"request":"coder@<REDACTED>'s password:"}
[15:40:54.921] Showing password prompt
[15:40:54.922] Listening for interwindow password on /run/user/1000/vscode-ssh-askpass-a5f6e61d5f40ae828442b137dc70f507afe0a34e.sock
[15:40:54.922] Writing password prompt to globalState
[15:40:59.562] Got password response
[15:40:59.562] Interactor gave response: ****************
[15:40:59.563] Cleaning up other-window auth server
[15:40:59.673] stderr> Authenticated to <REDACTED> ([<REDACTED>]:<REDACTED>) using "password".
[15:40:59.858] > Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-94-generic x86_64)
> 
>  * Documentation:  https://help.ubuntu.com
>  * Management:     https://landscape.canonical.com
>  * Support:        https://ubuntu.com/advantage
> 
> This system has been minimized by removing packages and content that are
> not required on a system that users do not log into.
> 
> To restore this content, you can run the 'unminimize' command.
[15:40:59.953] > ready: 48981f887e83
[15:41:00.050] > Linux 5.4.0-94-generic #106-Ubuntu SMP Thu Jan 6 23:58:14 UTC 2022
[15:41:00.050] Platform: linux
[15:41:00.139] > /bin/bash
[15:41:00.139] Shell: bash
[15:41:00.139] Creating bash subshell inside "bash"
[15:41:00.227] > bash version:  5.0.17(1)-release
[15:41:00.284] > bash version:  5.0.17(1)-release
[15:41:00.385] > 48981f887e83: running
[15:41:00.410] > Acquiring lock on /home/coder/.vscode-server/bin/74b1f979648cc44d385a2286793c226e611f59e7/vscode-remote-lock.coder.74b1f979648cc44d385a2286793c226e611f59e7

[Priya22]

In my case, it turned out to be a locking issue with flock on the server side. I could see several flock processes in the D state. Once the file server was rebooted it worked fine. Thanks!

[vinicius507]

In my case, it turned out to be a locking issue with flock on the server side. I could see several flock processes in the D state. Once the file server was rebooted it worked fine. Thanks!

In my case, there are no flock processes in the D state.

[eleanorjboyd]

In my case, it turned out to be a locking issue with flock on the server side. I could see several flock processes in the D state. Once the file server was rebooted it worked fine. Thanks!

Hello @Priya22! Thank you for this helpful comment, to clarify how did you know that the flock process was in the D state? This would be useful to understand from my team's end. Thanks!

[eleanorjboyd]

In my case, it turned out to be a locking issue with flock on the server side. I could see several flock processes in the D state. Once the file server was rebooted it worked fine. Thanks!

In my case, there are no flock processes in the D state.

hi @vinicius507, could you try a few items to see if these might help? On the command palette select kill remote server or try toggling the useLocalServer and useFlock settings. Thanks!

[Priya22]

In my case, it turned out to be a locking issue with flock on the server side. I could see several flock processes in the D state. Once the file server was rebooted it worked fine. Thanks!

Hello @Priya22! Thank you for this helpful comment, to clarify how did you know that the flock process was in the D state? This would be useful to understand from my team's end. Thanks!

Hi @eleanorjboyd, I just ran the top command on my server, and there were several flock processes (I'm guessing from all the times I tried to connect to the server with vscode) with the Status (S) column showing D.

[vinicius507]

In my case, it turned out to be a locking issue with flock on the server side. I could see several flock processes in the D state. Once the file server was rebooted it worked fine. Thanks!

In my case, there are no flock processes in the D state.

hi @vinicius507, could you try a few items to see if these might help? On the command palette select kill remote server or try toggling the useLocalServer and useFlock settings. Thanks!

Toggling either useLocalServer and/or useFlock does not solve the issue. Without flock the password prompt keeps showing up.

The problem seems to be with our NFS storage. I've tested using flock -w 40 --verbose -xn <fd> and the process hangs as soons as it is spawned with the D state.

[eleanorjboyd]

@vinicius507 Are you able to turn off flock and try connecting that way? If you run into an issue there you can share the logs and we could help debug from that direction but flock and NFS is out of our domain at this point. Thanks!

[Priya22]

Just wanted to add that in addition to turning off useFlock and useLocalServer, it helped me to turn set Local Server Download to always. Like others mentioned, maybe the download speed from the vscode website is affecting something.

[eleanorjboyd]

Hello! vinicius507 are you still experiencing this issue? If so please let me know your current status and the outcome of trying the steps in my Oct 13th comment. Thanks!

[github-actions[bot]]

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

[vinicius507]

Hello! vinicius507 are you still experiencing this issue? If so please let me know your current status and the outcome of trying the steps in my Oct 13th comment. Thanks!

Greetings, it worked, actually. Sorry, I forgot to answer.

[eleanorjboyd]

No worries! Glad it worked out, thanks!