vscode-remote-release icon indicating copy to clipboard operation
vscode-remote-release copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

Investigate using local server on remoteListenOnSocket

Open jasperhyp opened this issue 3 years ago • 24 comments

I was trying to get onto the second layer of an HPC after SSH-ing with some Slurm command (srun), as was generally described in #1722. I tried the newest RemoteCommand from the Pre-release version of Remote SSH (following instructions in this issue), but it seems not passing the RemoteCommand when connecting.

  • VSCode Version: 1.63
  • Local OS Version: Windows 10 21H2 (Release Preview)
  • Remote OS Version: CentOS Linux release 7.9.2009
  • Remote Extension/Connection Type: SSH
Logs 
[11:47:37.895] Log Level: 2
[11:47:37.897] [email protected]
[11:47:37.897] win32 x64
[11:47:37.915] SSH Resolver called for "ssh-remote+some.host.name", attempt 1
[11:47:37.915] "remote.SSH.useLocalServer": false
[11:47:37.916] "remote.SSH.showLoginTerminal": false
[11:47:37.916] "remote.SSH.remotePlatform": {"instance-1":"linux","some.host.name":"linux"}
[11:47:37.916] "remote.SSH.path": undefined
[11:47:37.916] "remote.SSH.configFile": undefined
[11:47:37.916] "remote.SSH.useFlock": true
[11:47:37.916] "remote.SSH.lockfilesInTmp": false
[11:47:37.916] "remote.SSH.localServerDownload": auto
[11:47:37.916] "remote.SSH.remoteServerListenOnSocket": true
[11:47:37.917] "remote.SSH.showLoginTerminal": false
[11:47:37.917] "remote.SSH.defaultExtensions": []
[11:47:37.917] "remote.SSH.loglevel": 2
[11:47:37.917] "remote.SSH.enableDynamicForwarding": true
[11:47:37.917] "remote.SSH.enableRemoteCommand": true
[11:47:37.917] "remote.SSH.serverPickPortsFromRange": {}
[11:47:37.917] "remote.SSH.serverInstallPath": {}
[11:47:37.918] SSH Resolver called for host: some.host.name
[11:47:37.918] Setting up SSH remote "some.host.name"
[11:47:37.922] Using commit id "899d46d82c4c95423fb7e10e68eba52050e30ba3" and quality "stable" for server
[11:47:37.925] Install and start server if needed
[11:47:37.927] Checking ssh with "ssh -V"
[11:47:37.961] > OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
(HERE) [11:47:37.964] Running script with connection command: ssh -T -D 8501 -o RemoteCommand=none "some.host.name" bash
[11:47:37.966] Terminal shell path: C:\Windows\System32\cmd.exe
[11:47:38.158] > ]0;C:\Windows\System32\cmd.exe 
[11:47:38.158] Got some output, clearing connection timeout
[11:47:38.409] > P
[11:47:38.446] > Password: 
[11:47:38.447] Showing password prompt
[11:47:43.428] Got password response
[11:47:43.429] "install" wrote data to terminal: "*******************"
[11:47:43.446] > 
[11:47:58.917] > 91a30fe8ff8e: running
[11:47:59.003] > Acquiring lock on /home/yeh803/.vscode-server/bin/899d46d82c4c95423fb7e10e68eba5
> 2050e30ba3/vscode-remote-lock.yeh803.899d46d82c4c95423fb7e10e68eba52050e30ba3   
[11:47:59.041] > Found existing installation at /home/yeh803/.vscode-server/bin/899d46d82c4c95423
> fb7e10e68eba52050e30ba3...
[11:47:59.075] > Checking /home/yeh803/.vscode-server/.899d46d82c4c95423fb7e10e68eba52050e30ba3.l
> og and /home/yeh803/.vscode-server/.899d46d82c4c95423fb7e10e68eba52050e30ba3.pid
>  for a running server
[11:47:59.084] > Looking for server with pid: 31076
[11:47:59.173] > Starting server with command... /home/yeh803/.vscode-server/bin/899d46d82c4c9542
> 3fb7e10e68eba52050e30ba3/server.sh --start-server --host=127.0.0.1 --enable-remo
> te-auto-shutdown   --socket-path=/run/user/184812/vscode-ssh-remote-server-sock-
> 1640018879  &> "/home/yeh803/.vscode-server/.899d46d82c4c95423fb7e10e68eba52050e
> 30ba3.log" < /dev/null
> printenv:
>     MANPATH=/n/app/lmod/lmod/share/man::
>     XDG_SESSION_ID=20830
>     rvm_bin_path=/usr/local/rvm/bin
>     GEM_HOME=/usr/local/rvm/gems/ruby-2.4.9      
>     SHELL=/bin/bash
>     IRBRC=/usr/local/rvm/rubies/ruby-2.4.9/.irbrc
>     MODULEPATH_ROOT=/n/app/lmod/lmod/modulefiles 
>     SSH_CLIENT=73.61.9.36 9459 22
>     CONDA_SHLVL=1
>     CONDA_PROMPT_MODIFIER=(base)
>     LMOD_PKG=/n/app/lmod/lmod
[11:47:59.181] > 
>     MY_RUBY_HOME=/usr/local/rvm/rubies/ruby-2.4.9
>     LMOD_VERSION=7.5.17
>     QT_GRAPHICSSYSTEM_CHECKED=1
>     USER=yeh803
>     LMOD_sys=Linux
>     CONDA_EXE=/n/app/conda2/bin/conda
>     rvm_path=/usr/local/rvm
>     VSCODE_AGENT_FOLDER=/home/yeh803/.vscode-server
>     _CE_CONDA=
>     rvm_prefix=/usr/local
>     MAIL=/var/mail/yeh803
>     PATH=/n/app/conda2/bin:/n/app/conda2/condabin:/usr/local/rvm/gems/ruby-2.4.9
> /bin:/usr/local/rvm/gems/ruby-2.4.9@global/bin:/usr/local/rvm/rubies/ruby-2.4.9/
> bin:/n/cluster/bin:/opt/singularity/bin:/usr/local/bin:/usr/bin:/opt/puppetlabs/
> bin:/usr/local/rvm/bin
>     SLURM_CONF=/etc/slurm/slurm.conf
>     CONDA_PREFIX=/n/app/conda2
>     LMOD_SETTARG_CMD=:
>     PWD=/home/yeh803
>     LANG=en_US.UTF-8
>     MODULEPATH=/n/app/lmod/lmod/modulefiles/Linux:/n/app/lmod/lmod/modulefiles/C
> ore
>     TZ=America/New_York
>     LMOD_CMD=/n/app/lmod/lmod/libexec/lmod
>     _CE_M=
>     HMS_CLUSTER=o2
>     KRB5CCNAME=FILE:/tmp/krb5cc_184812_zPUZlA
>     rvm_version=1.29.10 (latest)
>     SHLVL=3
>     HOME=/home/yeh803
>     BASH_ENV=/n/app/lmod/lmod/init/bash
>     SSH_ORIGINAL_COMMAND=bash
>     CONDA_PYTHON_EXE=/n/app/conda2/bin/python
>     LOGNAME=yeh803
>     GEM_PATH=/usr/local/rvm/gems/ruby-2.4.9:/usr/local/rvm/gems/ruby-2.4.9@globa
> l
>     SSH_CONNECTION=73.61.9.36 9459 134.174.159.26 22
>     MODULESHOME=/n/app/lmod/lmod
>     CONDA_DEFAULT_ENV=base
>     LESSOPEN=||/usr/bin/lesspipe.sh %s
>     SHARED_DATABASES=/n/groups/shared_databases
>     LMOD_FULL_SETTARG_SUPPORT=no
>     XDG_RUNTIME_DIR=/run/user/184812
>     LMOD_DIR=/n/app/lmod/lmod/libexec
>     RUBY_VERSION=ruby-2.4.9
>     BASH_FUNC_module()=() {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTAR
> G_CMD:-:} -s sh)
>     }
>     BASH_FUNC_ml()=() {  eval $($LMOD_DIR/ml_cmd "$@")
>     }
>     _=/usr/bin/printenv
[11:47:59.188] > 
[11:47:59.207] > Spawned remote server: 104641
> Waiting for server log...
[11:47:59.223] > 
[11:47:59.274] > Waiting for server log...
[11:47:59.374] > Waiting for server log...
[11:47:59.471] > Waiting for server log...
[11:47:59.571] > Waiting for server log...
[11:47:59.673] > Waiting for server log...
[11:47:59.773] > Waiting for server log...
[11:47:59.874] > Waiting for server log...
[11:47:59.972] > Waiting for server log...
[11:48:00.070] > Waiting for server log...
[11:48:00.172] > Waiting for server log...
[11:48:00.273] > Waiting for server log...
[11:48:00.373] > Waiting for server log...
[11:48:00.474] > Waiting for server log...
[11:48:00.575] > Waiting for server log...
[11:48:00.673] > Waiting for server log...
[11:48:00.709] > Waiting for server log...
[11:48:00.771] > Waiting for server log...
[11:48:00.881] >  
> *
> * Reminder: You may only use this software with Visual Studio family products,  
> * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057)
> *
> 
[11:48:00.906] > 
[11:48:00.974] > 91a30fe8ff8e: start
> SSH_AUTH_SOCK====
> DISPLAY====
> webUiAccessToken====
> listeningOn==/run/user/184812/vscode-ssh-remote-server-sock-1640018879==        
> osReleaseId==centos==
> arch==x86_64==
> tmpDir==/run/user/184812==
> platform==linux==
> unpackResult====
> didLocalDownload==0==
> downloadTime====
> installTime====
> extInstallTime====
> serverStartTime==1695==
> connectionToken==aa11111a-1aa1-111a-a111-11111a111a1a==
> 91a30fe8ff8e: end
[11:48:00.976] Received install output: 
SSH_AUTH_SOCK====
DISPLAY====
webUiAccessToken====
listeningOn==/run/user/184812/vscode-ssh-remote-server-sock-1640018879==        
osReleaseId==centos==
arch==x86_64==
tmpDir==/run/user/184812==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====
extInstallTime====
serverStartTime==1695==
connectionToken==aa11111a-1aa1-111a-a111-11111a111a1a==
[11:48:00.977] Remote server is listening on /run/user/184812/vscode-ssh-remote-server-sock-1640018879
[11:48:00.977] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"socketPath":"/run/user/184812/vscode-ssh-remote-server-sock-1640018879"},"osReleaseId":"centos","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/184812","platform":"linux","connectionToken":"aa11111a-1aa1-111a-a111-11111a111a1a"},"serverStartTime":1695,"installUnpackCode":""}
(HERE) [11:48:00.979] Running script with connection command: ssh -T -L 127.0.0.1:8507:/run/user/184812/vscode-ssh-remote-server-sock-1640018879 -o RemoteCommand=none "some.host.name" bash
[11:48:00.980] Spawning tunnel with: type "C:\Users\Yep\AppData\Local\Temp\vscode-linux-multi-line-command-some.host.name-223392234.sh" | ssh -T -L 127.0.0.1:8507:/run/user/184812/vscode-ssh-remote-server-sock-1640018879 -o RemoteCommand=none "some.host.name" bash
[11:48:00.981] Terminal shell path: C:\Windows\System32\cmd.exe
[11:48:00.995] > 
[11:48:01.171] > ]0;C:\Windows\System32\cmd.exe 
[11:48:01.538] > P
[11:48:01.822] > Password: 
[11:48:01.823] Showing password prompt
[11:48:06.095] Got password response
[11:48:06.095] "SSH Tunnel" wrote data to terminal: "*******************"
[11:48:06.127] > 
[11:48:14.043] > SSH_AUTH_SOCK=
> Connected to SSH Host - Please do not close this terminal
[11:48:14.044] Spawned SSH tunnel between local port 8507 and remote target /run/user/184812/vscode-ssh-remote-server-sock-1640018879
[11:48:14.044] Waiting for ssh tunnel to be ready
[11:48:14.045] Tunneled /run/user/184812/vscode-ssh-remote-server-sock-1640018879 to local port 8507
[11:48:14.045] Resolved "ssh-remote+some.host.name" to "127.0.0.1:8507"
[11:48:14.048] ------

Steps to Reproduce:

  1. Add "remote.SSH.enableRemoteCommand": true, "remote.SSH.useLocalServer": true, "remote.SSH.remoteServerListenOnSocket": true to settings.json. "remote.SSH.remoteServerListenOnSocket": true is a must because otherwise would result in Failed to set up socket for dynamic port forward to remote port 45824: Socket closed.

  2. SSH to some.host.name with the following ssh config:

Host some.host.name
    User some.user.name
    // ForwardAgent yes  // Seems this does not matter
    RemoteCommand srun -t 0-12:00 --pty -p gpu --gres=gpu:1 /bin/bash

Does this issue occur when you try this locally?: It's a connection issue.

jasperhyp avatar Dec 20 '21 20:12 jasperhyp

Hm and when you connect to this same host through PowerShell do you see your command in RemoteCommand being passed as expected?

tanhakabir avatar Jan 05 '22 09:01 tanhakabir

Hm and when you connect to this same host through PowerShell do you see your command in RemoteCommand being passed as expected?

Yes, when I use PowerShell with ssh host.name.in.the.ssh.config, this RemoteCommand is passed, as shown in the snapshot below.

image

Never mind the error message. It just showed the remote command has been passed.

jasperhyp avatar Jan 07 '22 16:01 jasperhyp

Well... maybe this error message is a problem even if the RemoteCommand can be passed in VSCode-Remote, since I am also getting into this error using ssh something 'srun something' command in MobaXterm (but np if I run srun something after ssh something to the login node).

However, it stays true that RemoteCommand (even as simple as ls) is not passed in VSCode, while both PowerShell and Moba pass it:

image

image

image

jasperhyp avatar Jan 07 '22 16:01 jasperhyp

Oh I missed this earlier. I see in your logs you have [11:47:37.915] "remote.SSH.useLocalServer": false

You need to have useLocalServer enabled to use RemoteCommand. That's why you see in the connection command RemoteCommand=none

tanhakabir avatar Jan 07 '22 19:01 tanhakabir

Oh I missed this earlier. I see in your logs you have [11:47:37.915] "remote.SSH.useLocalServer": false

You need to have useLocalServer enabled to use RemoteCommand. That's why you see in the connection command RemoteCommand=none

Oh looks like this parameter is set to false in ssh. But I did set the parameter to true in settings.json, and the box before Remote.SSH: Use Local Server is shown to be marked:

    "remote.SSH.useLocalServer": true,
    "remote.SSH.enableRemoteCommand": true,
    "terminal.integrated.inheritEnv": false,
    "remote.SSH.remoteServerListenOnSocket": true,
    "window.zoomLevel": 1,

image

jasperhyp avatar Jan 07 '22 22:01 jasperhyp

Try setting "remote.SSH.remoteServerListenOnSocket" to false.

They can't work in conjunction.

tanhakabir avatar Jan 07 '22 23:01 tanhakabir

  1. Failed to set up socket for dynamic port forward to remote port 45824: Socket closed

Thank you, I actually tried set "remote.SSH.remoteServerListenOnSocket": false and "remote.SSH.useLocalServer": true at the time I filed this issue, but it yielded an error Failed to set up socket for dynamic port forward to remote port XXXX: Socket closed. It's still the case now.

jasperhyp avatar Jan 08 '22 03:01 jasperhyp

Oh I see, hm that's unfortunate. Right now we can only support RemoteCommand when local server is enabled but for some machines that method doesn't work and remoteServerListenOnSocket is needed like your machine seems to. I'm not sure if there's a resolution for this.

tanhakabir avatar Jan 08 '22 07:01 tanhakabir

Oh I see, hm that's unfortunate. Right now we can only support RemoteCommand when local server is enabled but for some machines that method doesn't work and remoteServerListenOnSocket is needed like your machine seems to. I'm not sure if there's a resolution for this.

Got it, thanks! I tried again today and it just doesn't work. Maybe I'll just use VS Code to view the code but not run anything.

jasperhyp avatar Jan 10 '22 00:01 jasperhyp

Sorry about that! Do you need RemoteCommand to run/compile your code?

tanhakabir avatar Jan 10 '22 03:01 tanhakabir

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

github-actions[bot] avatar Jan 17 '22 12:01 github-actions[bot]

Sorry about that! Do you need RemoteCommand to run/compile your code?

Thanks for the follow-up. Have been out of work for a few days. I am mainly using Python so usually compilation is not needed. I just sometimes want to use .ipynb files to test some code. My current workflow is to use VS Code's Remote SSH for viewing & editing (because I love your great add-ons), and use Jupyter notebook & port-forwarding to test small code. That works despite a bit more windows.

jasperhyp avatar Jan 18 '22 05:01 jasperhyp

use Jupyter notebook & port-forwarding to test small code

Is this done locally on your machine or on a remote machine? I believe everything you do locally should be able to be done in your remote if you wish it have it running there. Maybe just not as a RemoteCommand that runs on connection to the remote.

tanhakabir avatar Jan 18 '22 17:01 tanhakabir

use Jupyter notebook & port-forwarding to test small code

Is this done locally on your machine or on a remote machine? I believe everything you do locally should be able to be done in your remote if you wish it have it running there. Maybe just not as a RemoteCommand that runs on connection to the remote.

I meant to say "open Jupyter Notebook on the remote machine and port forward to local machine with browser". I cannot do this with Remote SSH because I need to run RemoteCommand to get to the second (computing) layer of the HPC to run .ipynb files (since they are too resource-intensive for a login layer).

jasperhyp avatar Jan 19 '22 15:01 jasperhyp

Ahh I see. Thanks for explaining! Sorry about that! I'll explore alternatives to localServer with remoteListenOnSocket is needed.

tanhakabir avatar Jan 19 '22 18:01 tanhakabir

I face the same issue. I need to use slurm for ressource allocation because the login nodes are slow (they are login nodes after all) and cannot handle vscode server properly (ie most features are significanly slowed-down).

Standard ssh with port forwarding directly to the compute node doesn't works (ssh -L) but I can reverse forward from the compute to the login node (e.g. I can forward tensorboard running on the compute node)

my .ssh/config content (I need to use a jump node to get access to the login node)

Host zzz_vscode
     HostName zzz
     User yyy
     ProxyCommand XXX
     RequestTTY yes	 
     RemoteCommand srun -N1  --account=xxx --cpus-per-task=1 --pty bash
Vscode SSH extension logs 

[14:40:05.498] Log Level: 1
[14:40:05.499] [email protected]
[14:40:05.499] darwin arm64
[14:40:05.500] SSH Resolver called for "ssh-remote+zzz_vscode", attempt 1
[14:40:05.500] "remote.SSH.useLocalServer": true
[14:40:05.500] "remote.SSH.path": undefined
[14:40:05.500] "remote.SSH.configFile": undefined
[14:40:05.500] "remote.SSH.useFlock": true
[14:40:05.500] "remote.SSH.lockfilesInTmp": false
[14:40:05.501] "remote.SSH.localServerDownload": auto
[14:40:05.501] "remote.SSH.remoteServerListenOnSocket": false
[14:40:05.501] "remote.SSH.showLoginTerminal": false
[14:40:05.501] "remote.SSH.defaultExtensions": []
[14:40:05.502] "remote.SSH.loglevel": 1
[14:40:05.502] "remote.SSH.enableDynamicForwarding": true
[14:40:05.502] "remote.SSH.enableRemoteCommand": true
[14:40:05.502] "remote.SSH.serverPickPortsFromRange": {}
[14:40:05.502] "remote.SSH.serverInstallPath": {}
[14:40:05.508] SSH Resolver called for host: zzz_vscode
[14:40:05.508] Setting up SSH remote "zzz_vscode"
[14:40:05.510] Acquiring local install lock: /var/folders/17/tyfgx15d01l1fv0sxqgmftx80000gn/T/vscode-remote-ssh-db60dc3f-install.lock
[14:40:05.511] Looking for existing server data file at /Users/yyy/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-db60dc3f-62de8dc3f78b8bef1eb62deeb093164a8d76c254-0.73.2022020215/data.json
[14:40:05.511] Found local server running: {"remoteListeningOn":{"port":33051},"osReleaseId":"centos","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/8675","platform":"linux","connectionToken":"11aaaa11-1111-11aa-a111-11aa11111a11","pid":82322,"ipcHandlePath":"/var/folders/17/tyfgx15d01l1fv0sxqgmftx80000gn/T/vscode-ssh-askpass-d85a4b01a2eb3c8ceee690abf0c737630364345c.sock","socksPort":50839,"startupTime":1644143915576}
[14:40:05.513] Server delay-shutdown request failed: connect ECONNREFUSED /var/folders/17/tyfgx15d01l1fv0sxqgmftx80000gn/T/vscode-ssh-askpass-d85a4b01a2eb3c8ceee690abf0c737630364345c.sock
[14:40:05.513] Existing server's health check failed, will start a new one
[14:40:05.513] Using commit id "62de8dc3f78b8bef1eb62deeb093164a8d76c254" and quality "insider" for server
[14:40:05.515] Install and start server if needed
[14:40:05.518] PATH: /Users/yyy/.jenv/shims:/Users/yyy/.jenv/bin:/opt/homebrew/opt/rclone-mac/libexec/rclone:/Users/yyy/.miniconda3/condabin:/opt/local/bin:/opt/local/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion Tech Preview.app/Contents/Public:/usr/local/MacGPG2/bin:/opt/X11/bin:/Library/Apple/usr/bin:/opt/homebrew/opt/fzf/bin
[14:40:05.518] Checking ssh with "ssh -V"
[14:40:05.527] > OpenSSH_8.6p1, LibreSSL 2.8.3

[14:40:05.528] askpass server listening on /var/folders/17/tyfgx15d01l1fv0sxqgmftx80000gn/T/vscode-ssh-askpass-6b4f514209bbb52a2c2afe1e255de03f4a726354.sock
[14:40:05.528] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/17/tyfgx15d01l1fv0sxqgmftx80000gn/T/vscode-ssh-askpass-646a156a10d8d7b8547059b25e7c52b9a475015b.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","63353","-o","ConnectTimeout=15","zzz_vscode"],"serverDataFolderName":".vscode-server-insiders","dataFilePath":"/Users/yyy/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-db60dc3f-62de8dc3f78b8bef1eb62deeb093164a8d76c254-0.73.2022020215/data.json"}
[14:40:05.528] Local server env: {"DISPLAY":"/private/tmp/com.apple.launchd.j9qwA5HaxM/org.xquartz:0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/yyy/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.73.2022020215/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code - Insiders.app/Contents/MacOS/Electron","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"/Users/yyy/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.73.2022020215/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/17/tyfgx15d01l1fv0sxqgmftx80000gn/T/vscode-ssh-askpass-6b4f514209bbb52a2c2afe1e255de03f4a726354.sock"}
[14:40:05.529] Spawned 89305
[14:40:05.579] > local-server-1> Spawned ssh, pid=89315
[14:40:05.581] stderr> OpenSSH_8.6p1, LibreSSL 2.8.3
[14:40:05.581] stderr> debug1: Reading configuration data /Users/yyy/.ssh/config
[14:40:05.581] stderr> debug1: /Users/yyy/.ssh/config line 6: Applying options for *
[14:40:05.581] stderr> debug1: /Users/yyy/.ssh/config line 41: Applying options for zzz_vscode
[14:40:05.581] stderr> debug1: Reading configuration data /etc/ssh/ssh_config
[14:40:05.581] stderr> debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
[14:40:05.581] stderr> debug1: /etc/ssh/ssh_config line 54: Applying options for *
[14:40:05.582] stderr> debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
[14:40:05.582] stderr> debug1: auto-mux: Trying existing master
[14:40:05.582] stderr> debug1: Requesting forwarding of dynamic forward LOCALHOST:63353 -> *
[14:40:05.592] stderr> debug1: mux_client_request_session: master session id: 12
[14:40:06.196] stderr> manpath: can't set the locale; make sure $LC_* and $LANG are correct
[14:40:06.260] stderr> srun: error: ioctl(TIOCGWINSZ): Inappropriate ioctl for device
[14:40:06.260] stderr> srun: error: Not using a pseudo-terminal, disregarding --pty option
[14:40:06.421] > ready: b0826ae25f94
[14:40:06.434] > Linux 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021
[14:40:06.434] Platform: linux
[14:40:06.484] > b0826ae25f94: running
[14:40:06.524] > Acquiring lock on /home/yyy/.vscode-server-insiders/bin/62de8dc3f78b8bef1eb62deeb093164a8d76c254/vscode-remote-lock.yyy.62de8dc3f78b8bef1eb62deeb093164a8d76c254
[14:40:06.530] > Found existing installation at /home/yyy/.vscode-server-insiders/bin/62de8dc3f78b8bef1eb62deeb093164a8d76c254...
> Checking /home/yyy/.vscode-server-insiders/.62de8dc3f78b8bef1eb62deeb093164a8d76c254.log and /home/yyy/.vscode-server-insiders/.62de8dc3f78b8bef1eb62deeb093164a8d76c254.pid for a running server
[14:40:06.577] > Looking for server with pid: 10841
[14:40:06.627] > Starting server with command... /home/yyy/.vscode-server-insiders/bin/62de8dc3f78b8bef1eb62deeb093164a8d76c254/bin/code-server-insiders --start-server --host=127.0.0.1 --accept-server-license-terms --enable-remote-auto-shutdown --port=0    &> "/home/yyy/.vscode-server-insiders/.62de8dc3f78b8bef1eb62deeb093164a8d76c254.log" < /dev/null
> printenv:
[14:40:06.631] >     NVM_INC=/home/yyy/.nvm/versions/node/v14.18.1/include/node
>     SLURM_NODELIST=zzz-n5
>     SLURM_JOB_NAME=bash
>     XDG_SESSION_ID=549
>     SLURM_TOPOLOGY_ADDR=zzz-n5
>     SLURMD_NODENAME=zzz-n5
>     SELINUX_ROLE_REQUESTED=
>     SLURM_PRIO_PROCESS=0
>     NVM_CD_FLAGS=
>     SLURM_SRUN_COMM_PORT=62456
>     SHELL=/bin/bash
>     SLURM_JOB_QOS=normal
>     SSH_CLIENT=213.135.51.79 41156 22
>     SLURM_TOPOLOGY_ADDR_PATTERN=node
>     TMPDIR=/tmp
>     CONDA_SHLVL=0
>     SELINUX_USE_CURRENT_RANGE=
>     SLURM_CPU_BIND_VERBOSE=quiet
>     SLURM_CPU_BIND_LIST=0x000000002
>     HISTFILESIZE=10000
>     NVM_DIR=/home/yyy/.nvm
>     USER=yyy
>     SLURM_NNODES=1
>     CONDA_EXE=/home/yyy/.miniconda/bin/conda
>     SLURM_STEP_NUM_NODES=1
>     SRUN_DEBUG=3
>     SLURM_JOBID=51884
>     SLURM_NTASKS=1
>     SLURM_LAUNCH_NODE_IPADDR=172.24.40.5
>     SLURM_STEP_ID=0
>     VSCODE_AGENT_FOLDER=/home/yyy/.vscode-server-insiders
>     _CE_CONDA=
>     MAIL=/var/mail/yyy
>     PATH=/home/yyy/.nvm/versions/node/v14.18.1/bin:/home/yyy/.miniconda/condabin:/usr/local/bin:/usr/bin:/apps/icm-tools/bin
>     SLURM_TASKS_PER_NODE=1
>     SLURM_STEP_LAUNCHER_PORT=62456
>     SLURM_CONF=/usr/local/etc/slurm/slurm.conf
>     SLURM_WORKING_CLUSTER=zzz:zzz-mgmt2.icm.edu.pl:6817:8960:101
>     SLURM_CPUS_PER_TASK=1
>     SLURM_JOB_ID=51884
>     PWD=/lu/tetyda/home/yyy
>     SLURM_STEPID=0
>     SLURM_JOB_USER=yyy
>     SLURM_SRUN_COMM_HOST=172.24.40.5
>     SLURM_CPU_BIND_TYPE=mask_cpu:
>     LANG=en-US
>     MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles
>     SLURM_UMASK=0022
>     LOADEDMODULES=
>     SLURM_JOB_UID=8675
>     SLURM_NODEID=0
>     SELINUX_LEVEL_REQUESTED=
>     SLURM_SUBMIT_DIR=/lu/tetyda/home/yyy
>     SLURM_STEP_RESV_PORTS=12371-12372
>     SLURM_NPROCS=1
>     SLURM_TASK_PID=12591
>     SLURM_CPUS_ON_NODE=1
>     HISTCONTROL=ignorespace:erasedups
>     _CE_M=
>     SLURM_PROCID=0
>     SLURM_JOB_NODELIST=zzz-n5
>     SHLVL=3
>     HOME=/home/yyy
>     SLURM_LOCALID=0
>     SLURM_CLUSTER_NAME=zzz
>     SLURM_JOB_CPUS_PER_NODE=1
>     SLURM_JOB_GID=2000
>     SLURM_SUBMIT_HOST=zzz.icm.edu.pl
>     SLURM_GTIDS=0
>     SLURM_JOB_PARTITION=gpu
>     CONDA_PYTHON_EXE=/home/yyy/.miniconda/bin/python
>     LOGNAME=yyy
[14:40:06.633] >     SLURM_STEP_NUM_TASKS=1
>     SSH_CONNECTION=213.135.51.79 41156 172.24.40.5 22
>     SLURM_JOB_ACCOUNT=gr85-0
>     MODULESHOME=/usr/share/Modules
>     SLURM_JOB_NUM_NODES=1
>     NVM_BIN=/home/yyy/.nvm/versions/node/v14.18.1/bin
>     LESSOPEN=||/usr/bin/lesspipe.sh %s
>     SLURM_STEP_TASKS_PER_NODE=1
>     SLURM_STEP_NODELIST=zzz-n5
>     XDG_RUNTIME_DIR=/run/user/8675
>     SLURM_CPU_BIND=quiet,mask_cpu:0x000000002
>     BASH_FUNC_module()=() {  eval `/usr/bin/modulecmd bash $*`
>     }
>     _=/usr/bin/printenv
[14:40:06.652] > Spawned remote server: 12637
[14:40:06.663] > Waiting for server log...
[14:40:06.724] > Waiting for server log...
[14:40:06.798] > Waiting for server log...
[14:40:06.866] >  
[14:40:06.875] > *
> * Reminder: You may only use this software with Visual Studio family products,
> * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057)
> *
>  
[14:40:06.885] > b0826ae25f94: start
> SSH_AUTH_SOCK====
> DISPLAY====
> webUiAccessToken====
> listeningOn==41925==
> osReleaseId==centos==
> arch==x86_64==
> tmpDir==/run/user/8675==
> platform==linux==
> unpackResult====
> didLocalDownload==0==
> downloadTime====
> installTime====
> extInstallTime====
> serverStartTime==230==
> connectionToken==a1a1111a-1111-1a11-1aa1-11a1a11aa11a==
> b0826ae25f94: end
[14:40:06.885] Received install output: 
SSH_AUTH_SOCK====
DISPLAY====
webUiAccessToken====
listeningOn==41925==
osReleaseId==centos==
arch==x86_64==
tmpDir==/run/user/8675==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====
extInstallTime====
serverStartTime==230==
connectionToken==a1a1111a-1111-1a11-1aa1-11a1a11aa11a==

[14:40:06.886] Remote server is listening on 41925
[14:40:06.886] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":41925},"osReleaseId":"centos","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/8675","platform":"linux","connectionToken":"a1a1111a-1111-1a11-1aa1-11a1a11aa11a"},"serverStartTime":230,"installUnpackCode":""}
[14:40:06.887] Persisting server connection details to /Users/yyy/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-db60dc3f-62de8dc3f78b8bef1eb62deeb093164a8d76c254-0.73.2022020215/data.json
[14:40:06.890] Starting forwarding server. localPort 63357 -> socksPort 63353 -> remotePort 41925
[14:40:06.890] Forwarding server listening on 63357
[14:40:06.890] Waiting for ssh tunnel to be ready
[14:40:06.891] Tunneled 41925 to local port 63357
[14:40:06.891] Resolved "ssh-remote+zzz_vscode" to "127.0.0.1:63357"
[14:40:06.892] Updating terminal environments: {}
[14:40:06.892] [Forwarding server 63357] Got connection 0
[14:40:06.898] TELEMETRY: {"eventName":"resolver","properties":{"osReleaseId":"centos","arch":"x86_64","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","remoteInConfigFile":"1","gotUnrecognizedPrompt":"0","dynamicForwarding":"1","localServer":"1","didLocalDownload":"0","installUnpackCode":"0","outcome":"success"},"measures":{"resolveAttempts":1,"retries":1,"timing.totalResolveTime":1398,"timing.preSshTime":28,"timing.establishSshTime":956,"timing.scriptTime":1358,"timing.serverStartTime":230}}
[14:40:06.901] ------




[14:40:06.913] Failed to set up socket for dynamic port forward to remote port 41925: Socket closed. Is the remote port correct?
[14:40:06.916] [Forwarding server 63357] Got connection 1
[14:40:06.926] Failed to set up socket for dynamic port forward to remote port 41925: Socket closed. Is the remote port correct?

ilyasdc avatar Feb 06 '22 13:02 ilyasdc

Ahh I see. Thanks for explaining! Sorry about that! I'll explore alternatives to localServer with remoteListenOnSocket is needed.

@tanhakabir did you had a chance to look at it and pinpoint the issue? would be really cool to just be able to use: RemoteCommand srun --pty --gres=gpu:1 /bin/bash and let vscode figure out which ports to forward/listen to, automatically.

tzom avatar Mar 10 '22 19:03 tzom

Greeting! I just found this Issue while I tried to achieve something similar (Running VSCode on a Slurm Node with singularity, Using RemoteCommand and LocalServer) and I am facing the exact same issue as @ilyasdc. The output of the install procedure looks almost identical, with the final error

Failed to set up socket for dynamic port forward to remote port

Can I somehow help with this Issue, maybe by providing Logfiles or similar? This feature would greatly benefit the research at my facility, so I would be very glad to help.

schroeni96 avatar Mar 14 '22 16:03 schroeni96

  1. Failed to set up socket for dynamic port forward to remote port 45824: Socket closed

Thank you, I actually tried set "remote.SSH.remoteServerListenOnSocket": false and "remote.SSH.useLocalServer": true at the time I filed this issue, but it yielded an error Failed to set up socket for dynamic port forward to remote port XXXX: Socket closed. It's still the case now.

I'm seeing similar forwarding errors when using the RemoteCommand when trying to run the allocation through the command. So, same behavior as the one reported by @jasperhyp and @ilyasdc .

I can, however, execute commands directly on the login node.

It seems that something with the jump on the resource allocation (srun or salloc) messes up the setup from VSCode.

Happy to debug in case you need some logs.

adinriv avatar May 03 '22 07:05 adinriv

Workaround from a very clever friend (credits to Gwen, whose GitHub id I don't know about):

For those who only need to srun to the compute node, try first srun a compute node in server using anything else and then use ProxyJump command in config.ssh, e.g.

Host login-node
    HostName login-node.edu
    User hello

Host compute-node
    HostName %h
    ProxyJump login-node
    User hello

The assumption here is that you can use ssh compute-node in your server once you log onto the login node. Works for my case.

Also see here

jasperhyp avatar May 17 '22 01:05 jasperhyp

Workaround from a very clever friend (credits to Gwen, whose GitHub id I don't know about):

For those who only need to srun to the compute node, try first srun a compute node in server using anything else and then use ProxyJump command in config.ssh, e.g.

Host login-node
    HostName login-node.edu
    User hello

Host compute-node
    HostName %h
    ProxyJump login-node
    User hello

The assumption here is that you can use ssh compute-node in your server once you log onto the login node. Works for my case.

Also see here

If only it was as imple as that ... I cannot directly ssh to the compute node !

ilyasdc avatar May 17 '22 06:05 ilyasdc

I'm also facing the same usecase as @ilyasdc, but using a TORQUE instead of SLURM. Of course, the workaround proposed by @jasperhyp does not really work on batch systems as SSHing into compute nodes is forbidden. To me, it seems that the extension is ignoring the RequestTTY yes because in a normal terminal it works just fine. My config is:

Host login-interactive
  HostName ssh.login-node.edu
  User myuser
  RemoteCommand  qsub -I -W group_list=GROUP -A GROUP -l nodes=1:ppn=1,mem=1g,walltime=01:00:00
  RequestTTY force

Excerpt of output:

Authenticated to ssh.login-node.edu ([IP]:22).
qsub:	standard input and output must be a terminal for 
	interactive job submission

saiden89 avatar May 20 '22 13:05 saiden89

hi everyone :) any news on this issue? would be really helpful to have RemoteCommand+Slurm working together...

tzom avatar Sep 07 '22 09:09 tzom

My team and I are also desperately waiting for a solution to this. Any updates on the progress? Thanks!

PascalIversen avatar Sep 08 '22 15:09 PascalIversen

Bumping this up

alex-necula avatar Oct 11 '22 05:10 alex-necula

Hello! Could the people still interest in the thread outline their exact use case for this need. This would be a complicated fix so we are trying to determine if there are any solutions this problem that might be more successful and quicker. Thanks!

eleanorjboyd avatar Dec 08 '22 18:12 eleanorjboyd

Hi @eleanorjboyd ,

the original post describes it best (first post in this issue) https://github.com/microsoft/vscode-remote-release/issues/6086#issue-1085195764

The overall goal is to run the vscode-server on a compute node, e.g. in a computer cluster. Specifically, with RemoteCommand in the ssh config file we try to allocate a resource (via slurm or any other job scheduler) and let vscode-server run on the allocated node:

RemoteCommand srun --mem=16G -c=1 --time=08:00:00 --pty bash

This is crucial because in many compute clusters it is not allowed to run something like vscode-server on the head node itself. So RemoteCommand would solve this as it does run the slurm command first and then starts the vscode-server on the allocated resource. However, RemoteCommand does not exactly behave like it is supposed to. This issue #6086 narrows it down to the settings ("remoteServerListenOnSocket", "useLocalServer", etc.) that may interfere with RemoteCommand. Also, this issue is related to this issue #1722

hope it helps. Looking forward to having RemoteCommand properly working in combination with a compute cluster.

tzom avatar Dec 12 '22 10:12 tzom

Hi @eleanorjboyd , I believe there still should be interest.

The proposed solution is to run code-server on a remote machine we need to connect. In our case, it means scheduling a job via cluster management software (e.g. slurm) to run code-server binary on a computational node, and then get the stdout from code-server with connection details (pin code). Then use VSCode "Remote - Tunnels" plugin to connect to that code-server instance.

Downside of the proposed solution is: It uses Microsoft infrastructure to tunnel the traffic between Remote Tunnel plugin and code-server instance. If everything is in the same local net, this shouldn't be needed.

With "Remote - SSH" plugin the issue is the following. It allows to customize RemoteCommand, and we can use a customized RemoteCommand to run 'Code' on a computational node (instead of the node we established ssh connection to) and also to set up our own transparent tunneling between the computational node and the slurm control node, so that the 'Code' running on a computational node could be contacted via socket on a control node. However, we don't have a way for the Visual Studio Code (which is being run on a local PC) to use that particular socket instead of the one it thinks it have to use.

Hope I described the use case. If it's still not very clear, feel free to engage someone technical from your company (I'm sorry if it sounds rude, I just don't want this comment to be sort of useless or lost, and don't know how to express it better in words).

ekungurov avatar Dec 12 '22 12:12 ekungurov

Thanks for the explanations but I still don't fully understand the situation.

It sounds like this is issue essentially about using remoteServerListenOnSocket while also using useLocalServer so that RemoteCommand will work. That's probably just not going to work, due to the constraints on how things work that come from ssh. In that case, I wonder whether dropping remoteServerListenOnSocket is a solution for you. If there's a concern about security in having the server listen to a port, we have protections around access to the server's port that prevent other users from connecting to it.

Besides that, it sounds like what @ekungurov is saying is that using RemoteCommand for the slurm setup is not sufficient, and I think #1722 essentially covers that issue. I don't really understand it though-

can use a customized RemoteCommand to run 'Code' on a computational node (instead of the node we established ssh connection to) and also to set up our own transparent tunneling between the computational node and the slurm control node, so that the 'Code' running on a computational node could be contacted via socket on a control node. However, we don't have a way for the Visual Studio Code (which is being run on a local PC) to use that particular socket instead of the one it thinks it have to use.

With the RemoteCommand, wouldn't that mean that Remote-SSH connects and runs its setup script in the context of the compute node, and forwards our server's port/socket from the compute node to the local machine? Why do you say you have to set up your own forwarding from the login node?

roblourens avatar Dec 12 '22 22:12 roblourens

@roblourens on HPC users usually don't ssh directly to a compute node to launch a task. Instead, they ssh to a control node (login node) and then submit tasks to the cluster. Cluster mgmt software automatically choose a compute node (worker node) from the pool of compute nodes which currently have requested resources available.

That's why in #1722 a solution from @samfux84 (edited: this one https://github.com/microsoft/vscode-remote-release/issues/1722#issuecomment-837902373) use ssh -L for tunneling, and solution from @simonbyrne (this one https://github.com/microsoft/vscode-remote-release/issues/1722#issuecomment-1184705403) uses nc I believe for the same purpose.

ekungurov avatar Dec 13 '22 16:12 ekungurov