🐛 Connect to cloudflared ssh tunnel timeout with vscode

[n0k0m3]

Describe the bug Using default generated ssh config + short-lived certs by cloudflared, I was able to ssh using CLI normally. However using vscode to connect to the same hostname will timed out.

To Reproduce 0. Have a SSH + Short-lived cert setup on CFZT dashboard

1. Configure ssh config using cloudflared access ssh-config --hostname <hostname> --short-lived-cert
2. Use VSCode Remote SSH extension to connect to host using <username>@<hostname>
3. Error occurs

Expected behavior VSCode will connect and run normally with cloudflared access tunnel with short-lived certs

Environment and versions

• OS: Linux (Ubuntu Server, Arch Desktop)
• Architecture: AMD64
• Version: 2022.8.0

Logs and errors Client side log on vscode

[20:20:29.052] Spawned 32316
[20:20:29.132] > local-server-1> Spawned ssh, pid=32324
[20:20:29.134] stderr> OpenSSH_9.0p1, OpenSSL 1.1.1q  5 Jul 2022
[20:20:33.579] stderr> Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.13.0-28-generic x86_64)
[20:20:33.579] stderr> 
[20:20:33.579] stderr>  * Documentation:  https://help.ubuntu.com
[20:20:33.580] stderr>  * Management:     https://landscape.canonical.com
[20:20:33.580] stderr>  * Support:        https://ubuntu.com/advantage
[20:20:33.580] stderr> 
[20:20:33.580] stderr>   System information as of Fri 19 Aug 2022 12:20:33 AM UTC
[20:20:33.580] stderr> 
[20:20:33.580] stderr>   System load:                      0.06
[20:20:33.580] stderr>   Usage of /:                       21.1% of 936.92GB
[20:20:33.581] stderr>   Memory usage:                     3%
[20:20:33.581] stderr>   Swap usage:                       0%
[20:20:33.581] stderr>   Temperature:                      53.0 C
[20:20:33.581] stderr>   Processes:                        344
[20:20:33.581] stderr>   Users logged in:                  1
[20:20:33.581] stderr>   IPv4 address for stuffs
[20:20:33.582] stderr> 
[20:20:33.582] stderr>  * Super-optimized for small spaces - read how we shrank the memory
[20:20:33.582] stderr>    footprint of MicroK8s to make it the smallest full K8s around.
[20:20:33.583] stderr> 
[20:20:33.583] stderr>    https://ubuntu.com/blog/microk8s-memory-optimisation
[20:20:33.583] stderr> 
[20:20:33.583] stderr> 1 device has a firmware upgrade available.
[20:20:33.583] stderr> Run `fwupdmgr get-upgrades` for more information.
[20:20:33.583] stderr> 
[20:20:33.583] stderr> 
[20:20:33.584] stderr> 512 updates can be applied immediately.
[20:20:33.584] stderr> 277 of these updates are standard security updates.
[20:20:33.584] stderr> To see these additional updates run: apt list --upgradable
[20:20:33.584] stderr> 
[20:20:33.584] stderr> New release '22.04.1 LTS' available.
[20:20:33.584] stderr> Run 'do-release-upgrade' to upgrade to it.
[20:20:33.585] stderr> 
[20:20:33.585] stderr> Your Hardware Enablement Stack (HWE) is supported until April 2025.
[20:20:33.585] stderr> 
[20:20:33.585] stderr> Last login: Fri Aug 19 00:16:26 2022 from 127.0.0.1
[20:20:34.305] stderr> (base) username@hostname:~$ 
[20:20:44.144] stderr> Connection timed out during banner exchange ### (timeout after 10 secs, increased to 60 still the same)
[20:20:44.144] stderr> Connection to UNKNOWN port 65535 timed out
[20:20:44.145] > local-server-1> ssh child died, shutting down
[20:20:44.148] Local server exit: 0

Server side sshd completed (obviously from the log above after "Last Login" line which proves that ssh connection was achieved

Aug 19 00:21:31 username sshd[106223]: Accepted publickey for username from 127.0.0.1 port 34054 ssh2: <cert stuffs>
Aug 19 00:21:31 username sshd[106223]: pam_unix(sshd:session): session opened for user username by (uid=0)

Additional context I think the problem is VSCode expects the output of tty to be in stdout not stderr and was waiting for login completion.

Edit: I think this is the problematic line in config: For Host <FQDN>:

ProxyCommand bash -c '/usr/bin/cloudflared access ssh-gen --hostname %h; ssh -tt %r@cfpipe-<FQDN> >&2 <&1'

and for Host cfpipe-<FQDN>

ProxyCommand /usr/bin/cloudflared access ssh --hostname %h

Can we somehow pipe the 2nd one to stderr and let the first one take in stderr and pipe to stdout?

Edit 2: tried that, it breaks ssh

[n0k0m3]

It has been ~~3~~ 18 days, as I've seen other issues they'll typically be address in less time. Is there any update on this from cloudflare team?

[benlimpa]

I'm not sure if your situation is the same as mine, but I had the exact same symptoms. The problem for me turned out to be related to using fish as my login shell on the remote machine (see https://github.com/microsoft/vscode-remote-release/issues/2509). I fixed it by switching my default shell back to bash.

[sudarshan-reddy]

Sorry for the late response. But @benlimpa beat us to it. This is most likely your shell’s doing. Can you give it a try and let us know if you still are having problems?

[n0k0m3]

Resolved with https://github.com/cloudflare/cloudflared/issues/574#issuecomment-1143626460, please update cloudflared config gen with it.

Also, the solution works for Windows as well as Linux