vscode-remote-release
vscode-remote-release copied to clipboard
microsoft
"Connecting with SSH timed out" when Fish/tcsh/csh shell is used
workaround: in preferences, set remote.SSH.useLocalServer to true and remote.SSH.remotePlatform like so:
"remote.SSH.remotePlatform": {
<hostname1>: "linux",
<hostname2>: "linux"
},
~~workaround: Downgrading this extension to 0.49.0 fixes the issue (although it appears to break the "recent connections" UI). Extensions menu -> Right-click on "Remote - SSH" -> "Install Another Version" -> 0.49.0~~
This is a regression on #33, started after I upgraded to VScode 1.43.0/extension 0.50.0.
- VSCode Version: 1.43.0
- Local OS Version: macOS 10.14.6
- Remote OS Version: Ubuntu 18.04.3 LTS
- Remote Extension/Connection Type: SSH
Steps to Reproduce:
- Set bash to default shell
- Connect with VSCode remote. Works as expected.
- Set default remote shell as Fish
- Connecting with VSCode times out.
This was observed on two different servers
Does this issue occur when you try this locally?: N/A Does this issue occur when you try this locally and all extensions are disabled?: N/A
[11:40:01.637] Log Level: 2
[11:40:01.639] [email protected]
[11:40:01.639] darwin x64
[11:40:01.640] SSH Resolver called for "ssh-remote+anton", attempt 1
[11:40:01.640] SSH Resolver called for host: anton
[11:40:01.640] Setting up SSH remote "anton"
[11:40:01.643] Acquiring local install lock: /var/folders/y0/x378plh96hndqvy04th4v7880000gn/T/vscode-remote-ssh-anton-install.lock
[11:40:01.654] Looking for existing server data file at /Users/tal/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-anton-78a4c91400152c0f27ba4d363eb56d2835f9903a/data.json
[11:40:01.655] Using commit id "78a4c91400152c0f27ba4d363eb56d2835f9903a" and quality "stable" for server
[11:40:01.656] Install and start server if needed
[11:40:01.659] Checking ssh with "ssh -V"
[11:40:01.673] > OpenSSH_7.9p1, LibreSSL 2.7.3
[11:40:01.676] askpass server listening on /var/folders/y0/x378plh96hndqvy04th4v7880000gn/T/vscode-ssh-askpass-298a99f5c46843bde64ed3c630b85b9c9c6619df.sock
[11:40:01.677] Spawning local server with {"ipcHandlePath":"/var/folders/y0/x378plh96hndqvy04th4v7880000gn/T/vscode-ssh-askpass-42adc0358caea0dce208ba088ea9f67fc0396523.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","56862","-o","ConnectTimeout=30","-o","RemoteCommand=none","anton"],"dataFilePath":"/Users/tal/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-anton-78a4c91400152c0f27ba4d363eb56d2835f9903a/data.json"}
[11:40:01.677] Local server env: {"DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/tal/.vscode/extensions/ms-vscode-remote.remote-ssh-0.50.0/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)","VSCODE_SSH_ASKPASS_MAIN":"/Users/tal/.vscode/extensions/ms-vscode-remote.remote-ssh-0.50.0/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/y0/x378plh96hndqvy04th4v7880000gn/T/vscode-ssh-askpass-298a99f5c46843bde64ed3c630b85b9c9c6619df.sock"}
[11:40:01.679] Spawned 93505
[11:40:01.793] > local-server> Spawned ssh: 93506
[11:40:01.803] stderr> OpenSSH_7.9p1, LibreSSL 2.7.3
[11:40:03.569] stderr> debug1: Server host key: ecdsa-sha2-nistp256 SHA256:bFQr5QDzdBZHWOYB8GrdsncBvTdS47Ebd7x5ZKg/GKA
[11:40:04.998] stderr> Authenticated to <hidden> ([<hidden>]:22).
[11:40:05.920] > Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 5.0.0-37-generic x86_64)
>
> * Documentation: https://help.ubuntu.com
> * Management: https://landscape.canonical.com
> * Support: https://ubuntu.com/advantage
>
> * Latest Kubernetes 1.18 beta is now available for your laptop, NUC, cloud
> instance or Raspberry Pi, with automatic updates to the final GA release.
>
> sudo snap install microk8s --channel=1.18/beta --classic
>
> * Multipass 1.1 adds proxy support for developers behind enterprise
> firewalls. Rapid prototyping for cloud operations just got easier.
>
> https://multipass.run/
>
> * Canonical Livepatch is available for installation.
> - Reduce system reboots and improve kernel security. Activate at:
> https://ubuntu.com/livepatch
>
> 174 packages can be updated.
> 0 updates are security updates.
>
> Your Hardware Enablement Stack (HWE) is supported until April 2023.
> *** System restart required ***
[11:40:33.680] Terminating local server
[11:40:33.681] Resolver error: Connecting with SSH timed out
[11:40:33.685] ------
[11:40:33.687] Local server exit: 15
I can confirm, using fish shell, hell broke loose.
At firsh ssh-remote was complaining about requiring a non-windows ssh to work. I throught that was the issue but after installing git with ssh, the problem did not go away.
I just downgraded to 0.49, as suggested by @tals, and it works again.
I can confirm, using fish shell, hell broke loose.
At firsh ssh-remote was complaining about requiring a non-windows ssh to work. I throught that was the issue but after installing git with ssh, the problem did not go away.
I just downgraded to 0.49, as suggested by @tals, and it works again.
How can it downgrade to 0.49 ?
How can it downgrade to 0.49 ?
@qhhonx In the extensions menu, search for the remote-ssh extension and in the bottom right of the ssh-remote item, you will see a cog icon. Press it and you will see a menu with an option to install other versions.
How can it downgrade to 0.49 ?
@qhhonx In the extensions menu, search for the remote-ssh extension and in the bottom right of the ssh-remote item, you will see a cog icon. Press it and you will see a menu with an option to install other versions.
Big thanks for you, it works again as expected.
As a workaround for 0.50, you could try setting "remote.SSH.useLocalServer": false, and let me know whether that helps.
As a workaround for 0.50, you could try setting
"remote.SSH.useLocalServer": false, and let me know whether that helps.
yes, setting useLocalServer to false solves the issue.
As a workaround for 0.50, you could try setting
"remote.SSH.useLocalServer": false, and let me know whether that helps.
Can confirm this works
same problem here, was using fish for months without problems, suddenly failed (macOS 13.0.4), workaround also works here.
Sorry for the trouble everyone. I can't figure out what fish is doing here to break this. I just spawn an ssh process and pipe a script into stdin, but any command sent that way is simply ignored when fish is used.
So I tried the other fix: remote.SSH.useLocalServer=false and it fixes it for me too. I guess there is already an issue about local server enabled automatically so perhaps this issue is duplicated.
Sorry for the trouble everyone. I can't figure out what fish is doing here to break this. I just spawn an ssh process and pipe a script into stdin, but any command sent that way is simply ignored when fish is used.
@roblourens huh. judging by the other bugs here, it seems to happen with other shells like tcsh, so maybe its not fish specific.
After some experiments, I can see that fish wont process input being written to ssh if theres no terminal attached.
const childProcess = require('child_process')
const process = require('process')
const myServer = 'mg'
const p = childProcess.spawn(
'ssh',
// this wont print "hi there"
// ["-v","-T", "-o","ConnectTimeout=30",myServer],
// this will. note the "-tt" to attach a pseudo tty
["-v","-tt", "-o","ConnectTimeout=30",myServer],
{ stdio: 'pipe'}
)
p.stdin.write('echo "hi there!"\n')
p.stdout.on('data', e => {
process.stdout.write(e)
})
p.stderr.on('data', e=> {
process.stderr.write(e)
})
The reason it works without the local server is because the input stream terminates. You can see it by doing:
# this will print "hi"
$ echo "echo hi" | ssh -T mg
# this wont print anything
$ yes "echo hi" | ssh -T mg
# This print "hi"s forever
$ yes "echo hi" | ssh -T mg bash
# So will this (plus some shell prompts)
$ yes "echo hi" | ssh -tt mg
Sorry for the trouble everyone. I can't figure out what fish is doing here to break this. I just spawn an ssh process and pipe a script into stdin, but any command sent that way is simply ignored when fish is used.
Hey @roblourens, I think I found a workaround that allows the server to continue running.
In getSshConnectionArgs(), simply append "bash" to the ssh command, which skips fish.
I patched the extension locally and it seems to work. The integrated terminal still works.
(I couldn't find the source code on github. Would've opened a PR otherwise)
Yes, that does work. However, I want to be able to attach to windows, mac, or linux remotes without knowing what platform they are ahead of time, so I can't set bash as an initial command.
Thanks for your investigation, I have a nearly identical script 😁 except that I had tried -t and -ttt (thinking of -vvv) neither of which work, and -tt actually does. It's very awkward that it causes the input and a prompt to be echoed. And I'm nervous about introducing other issues by trying it in general. It might be safer to just say that people using fish have to set a setting (either to disable local server mode or to do opt-into the -tt flag). I get a mess of escape characters, and I will have to strip tab characters from the install script because they trigger actual tab completion. So I'm not looking forward to the weird issues that might come up if I do this for every connection. But this is a good hint about a possible path forward, thanks!
@roblourens ah I see.
You could run a probe for the platform in a preflight call. This is stable, because stdin would get closed as soon as uname -rsv gets dispatched by probeServerPlatform(), similarly to how it is when useLocalServer is off.
Once you know you're not on windows, you can go ahead with the bash bit (the mainScriptWrapper bit is simply a bash script, and its always used if you're not running windows).
The main issue I see with this is that the preflight would require an extra askpass for people without ssh keys, which is annoying to the user, but I think the useLocalServer=off bit has the same problem.
In theory once you've bootstrapped the remote machine, you can have a remote node agent that binds on a local port on the remote machine. localServer then runs it together with SSH port forwarding, and you use this ssh tunnel to send commands to the remote server, which runs them.
The main issue I see with this is that the preflight would require an extra askpass for people without ssh keys, which is annoying to the user, but I think the useLocalServer=off bit has the same problem.
Many things would be simpler if I could run multiple ssh commands, but I have managed to keep it to a single ssh connection at least per window, and I don't want to backtrack on that since many people use passwords, 2fa, etc which are very annoying to enter multiple times.
In theory once you've bootstrapped the remote machine, you can have a remote node agent that binds on a local port on the remote machine. localServer then runs it together with SSH port forwarding, and you use this ssh tunnel to send commands to the remote server, which runs them.
Well, this is exactly what we are doing, but for every connection we still have to run a script to determine whether the remote server is already running, or needs to be updated, or which port it is listening on, etc.
Many things would be simpler if I could run multiple ssh commands, but I have managed to keep it to a single ssh connection at least per window, and I don't want to backtrack on that since many people use passwords, 2fa, etc which are very annoying to enter multiple times.
@roblourens Makes sense. That's tricky! My understanding is that this was the case before useLocalServer was enabled by default in 0.50 -- you had to do at least two ssh connections: the preflight one and then the bootstrap one, so backtracking from that would lose some of the advantages of localServer.
How do you feel about caching the platform information, so its amortized?
I wonder if you could reliably infer if you're dealing with windows vs linux based on stuff coming back when you're doing the ssh handshake.
For example if I run ssh -T -v <server>, I get some server platform information back:
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
Here I can tell this is ubuntu linux before any command has been issued.
I don't have a windows machine available, but maybe it returns similar information so you could fingerprint it (and might be easier to fingerprint than linux haha)
Same issue here (macOS Catalina locally, some flavor of Ubuntu with a reasonably modern Fish remote). Downgrading fixed it, but setting "remote.SSH.useLocalServer" to false did not.
In fact, that was even worse. Somehow instead of just hanging, disabling it resulted in a really cool crash-loopy thing where it kept trying to connect and failing as fast as it could start, popping terminal windows open and closed.
@tals, without useLocalServer we still do one connection, but we have to know the platform ahead of time, assuming Linux unless the windowsRemotes setting contains the current host.
Connecting multiple times the first time and caching that info is a possibility but I really want the first experience to be good. Also considered just showing a popup for the user to select the platform, then keeping that info in a setting.
@bobthecow could you open a new issue and share which remote.SSH settings you have set, and the log from the Remote-SSH channel?
@roblourens I think asking for prompt on first use and then caching the information is a good compromise. As you are probably aware from the various bug-reports the current extension also breaks on (t)csh shells which are fairly common on RHEL/CentOS machines.
@roblourens The failure mode is almost exactly the same as with fish. Here is a similar experiment to @tals with tcsh being the login shell on the remote machine.
$ cat > test_tcsh.sh <<EOF
bash
function test_fn {
echo "> in test_fn"
}
test_fn
EOF
$ cat test_tcsh.sh | ssh -T remote_host
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
> in test_fn
function: Command not found.
}: Command not found.
test_fn: Command not found.
$ cat test_tcsh.sh | ssh -T remote_host bash
> in test_fn
Hm, I can repro it now, I don't know what I was doing wrong before. I agree that it's probably basically the same thing.
Set these config works for me:
"remote.SSH.showLoginTerminal": true,
"remote.SSH.useLocalServer": false,
VScode 1.43.2
ssh -V: OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
remote fish -v: fish, version 2.7.1
I'm see this issue as well when changing my local machine from a surface laptop to a Mac.
Still connecting to the same remote host that uses tcsh.
Default shell Mac is also tcsh.
Windows10 works fine however Mac is authenticating but breaks when seemingly sending bash commands to the remote tcsh shell
useLocalServer : false does help but you get stuck with this install terminal that seems to own the connection to the remote host
Also user experience is quite poor when needing to reconnect as you seem to get another install terminal
Is it possible the issue lies with the communication with the local server rather then the remote?
Yes I did also face same issue when fish the default shell in the remote shell(arm) doing the temporary workaround did work fine
