SSH Agent Forwarding not working with Remote SSH on MacOS (still a problem)

[mbmccoy]

The title is the same as #2671 because I'm having precisely the same problem, and that issue was closed without a resolution. In short, ssh agent forwarding does not work in VSCode, despite working just fine when I SSH in via the terminal. It seems to be due to a stale / cached SSH_AUTH_SOCK.

The workaround suggested in #2671 no longer works, but a minor modification does it for me. See below.

VSCode Version:

Version: 1.53.2
Commit: 622cb03f7e070a9670c94bae1a45d78d7181fbd4
Date: 2021-02-11T11:45:54.515Z
Electron: 11.2.1
Chrome: 87.0.4280.141
Node.js: 12.18.3
V8: 8.7.220.31-electron.0

OS: Darwin 20.6.0 Local OS Version: MacOS 11.6.1 Remote OS Version: Debian GNU/Linux 10 (buster) (GNU/Linux 4.19.0-18-cloud-amd64 x86_64) Remote Extension/Connection Type: SSH Steps to Reproduce:

1. set up ssh config with a host you want to forward your ssh agent keys to and set 'ForwardAgent yes'
2. Connect via terminal ssh and perform an operation requiring your forwarded key (in my case, ssh -T [email protected])
3. launch VSCode and validate that $SSH_AUTH_SOCK is set
4. Connect via Remote-SSH, then use the VSCode terminal to run ssh -T [email protected]. This time, I receive a permission denied error.

Does this issue occur when you try this locally?: NA Does this issue occur when you try this locally and all extensions are disabled?: NA

Possible cause

The environment variable SSH_AUTH_SOCK appears to be stale. A workaround is to run

export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1)

which sets the socket to the newest one. Note that this workaround is quite fragile, as I had to modify the previous workaround to make it work now.

[tanhakabir]

Could it be an issue with needing to set an IdentityAgent to the appropriate authentication agent since you said the current SSH_AUTH_SOCK env variable seems to be stale?

More details from the SSH Man(5) page:

Specifies the UNIX-domain socket used to communicate with the authentication agent.

This option overrides the SSH_AUTH_SOCK environment variable and can be used to select a specific agent. Setting the socket name to none disables the use of an authentication agent. If the string "SSH_AUTH_SOCK" is specified, the location of the socket will be read from the SSH_AUTH_SOCK environment variable. Otherwise if the specified value begins with a ‘$’ character, then it will be treated as an environment variable containing the location of the socket.

Arguments to IdentityAgent may use the tilde syntax to refer to a user's home directory, the tokens described in the TOKENS section and environment variables as described in the ENVIRONMENT VARIABLES section.

[mbmccoy]

Hmm.... I don't think that's the source of the problem. In particular, when I use ssh from the command line (using the same user/host that I use in VSC), I don't have this issue. I don't have any special settings for IdentityAgent in either my local or remote machine.

I'm often able to trigger the issue this way:

1. Open remote connection to server.
   • I can see that my identity is not appropriately forwarded via ssh -T [email protected] in VSCode terminal.
2. Run workaround in VSCode terminal: export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1).
   • Now ssh -T [email protected] succeeds.
3. Reload window (Developer: Reload Window).
   • Now ssh -T [email protected] fails.

Even weirder, after repeating steps 2 and 3 a few times, it's started using the correct socket after reload.

[roblourens]

What is $SSH_AUTH_SOCK before you run the workaround?

[mbmccoy]

I'm not able to trigger the issue right now, but $SSH_AUTH_SOCK was definitely set, and it looked structurally correct, like /tmp/ssh-cT3h6tFlau/agent.316, only the directory (and file) didn't exist.

[mbmccoy]

Ok, I'm able to reproduce now. These are all run from within the VSCode terminal. Here's the initial situation:

$ echo $SSH_AUTH_SOCK

/tmp/ssh-7G9TqKKzBp/agent.30503

$ ls -la /tmp/ssh*/**

srwxr-xr-x 1 xxx xxx 0 Feb  9 00:27 /tmp/ssh-9L8JuutTmn/agent.14198
srwxr-xr-x 1 xxx xxx 0 Feb  8 20:34 /tmp/ssh-jFxWY0hBUn/agent.16442

$ ssh -T [email protected]

[email protected]: Permission denied (publickey).

We can see that the SSH_AUTH_SOCK variable is not pointed at an extant ssh agent. Now we apply the workaround:

$ export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1)

$ ssh -T [email protected]

Hi mbmccoy! You've successfully authenticated, but GitHub does not provide shell access.

$ echo $SSH_AUTH_SOCK

/tmp/ssh-9L8JuutTmn/agent.14198

[roblourens]

Most likely you went through reconnection and are now connecting through a different ssh process, this means the original ssh auth sock is invalid. The extension tries to redirect SSH_AUTH_SOCK to a different path that we control, seems like that didn't work here. Do you have terminal.integrated.inheritEnv or remote.SSH.enableAgentForwarding disabled?

If not can you share the Remote-SSH log from the output panel?

[mbmccoy]

Here's the output for a run where I just had this issue:

https://gist.github.com/mbmccoy/331ba3674877b7c8769b482f78e467b7

Note that it does seem to have the correct agent in the logs (/tmp/ssh-vA5Po49GCF/agent.28474), but it was not correct in the integrated terminal because I had to update it using the fix mentioned above.

This makes me suspect that the terminal caches the SSH_AUTH_SOCK environment variable independent of the Remote SSH extension.

[roblourens]

Do you have terminal.integrated.inheritEnv or remote.SSH.enableAgentForwarding disabled?

Is this a terminal from a previous session that was reconnected, or a fresh terminal?

[github-actions[bot]]

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

[mbmccoy]

I suppose I dropped the ball on getting back to you on this. No, I don't have any settings that are related to the terminal or remote set in my settings.json.

The ssh command does include the -A flag.

Not sure if it matters, but I do log in to the machine from a terminal outside of VSCode (with agent forwarding enabled). It doesn't always seem to cause the issue.

I've updated to a recent version and I'm still occasionally seeing the issue.

Version: 1.66.0
Commit: e18005f0f1b33c29e81d732535d8c0e47cafb0b5
Date: 2022-03-30T05:50:04.419Z
Electron: 17.2.0
Chromium: 98.0.4758.109
Node.js: 16.13.0
V8: 9.8.177.11-electron.0
OS: Darwin x64 21.3.0

[roblourens]

Can you share the log from the Remote-SSH output channel?

Basically we should be trying to rewrite your SSH_AUTH_SOCK to something like /tmp/vscode-ssh-auth-sock-493210776 or /run/user/1000/vscode-ssh-auth-sock-493210776

[cdmistman]

I believe I'm experiencing the same issue

1. connect to remote host with ForwardAgent yes in ~/.ssh/config. terminal.integrated.inheritEnv and remote.SSH.enableAgentForwarding are both enabled in VSCode
2. verify a key is printed with ssh-add -L
3. echo $SSH_AUTH_SOCK
4. reload window
5. in the same terminal instance from steps 2 and 3, run ssh-add -L again - error code 2 Error connecting to agent: No such file or directory
6. again in the same terminal from step 5, echo $SSH_AUTH_SOCK - it prints the same value as step 3
7. open a new terminal (> Terminal: Create New Terminal) and execute ssh-add -L again - it should have the same result as step 2
8. in the same terminal from step 7, echo $SSH_AUTH_SOCK - it should print a different value from step 6

it seems terminals that are re-opened after a window reload don't get their SSH_AUTH_SOCK environment variable updated. The correct value is correctly set for all terminal instances that are created between window reloads

[mbmccoy]

Was able to trigger this just now. ~It may be related to the fact that I also connect (and forward my agent) from another terminal.~ (see note below) Here's the steps that I took:

1. Inside VSCode terminal connected to the remote, check that agent is forwarding:

    ssh -T [email protected]
    Hi mbmccoy! You've successfully authenticated, but GitHub does not provide shell access.
   

2. SSH into machine (forwarding the agent) from a non-VSCode terminal.

3. Reload window (Developer: Reload Window).

4. Check access:

    ssh -T [email protected]
    [email protected]: Permission denied (publickey).
   

Remote-SSH Logs here

NOTE (added in edit) This is also consistent with @cdmistman 's observation that

it seems terminals that are re-opened after a window reload don't get their SSH_AUTH_SOCK environment variable updated. The correct value is correctly set for all terminal instances that are created between window reloads

Opening a new terminal after reload gives access.

[davibe]

Same issue here, closing and recreating the terminal fixes it, but terminals cerated when vscode starts have a stale SSH_AUTH_SOCK

[snewell92]

The format of the fix above needed to be amended to match what I saw in my dev containers:

➜  ~ which ssh_vscode_fix
ssh_vscode_fix () {
	export SSH_AUTH_SOCK=$(ls -t /tmp/vscode-ssh-auth-*.sock | head -1)
}

[tamaszsellerp92]

hi I'm facing a similar issue with VScode latest I have the option ForwardAgent yes in my personal configuration file, however VSCode with Remote SSH seems to ignore it I can't for example clone any git repo or jump over to other hosts with this key, the SSH_AUTH_SOCK env. variable is totally missing when connecting from VSCode terminal. also it works from putty. I'm using windows (native win. SSH client, not WSL) sum: connecting to the linux machine with key works fine from VScode, but agent forwarding is not taken into account also using functions like git clone does not work from vscode, same issue - key is not forwarded. I can do it from terminal if I Set the SSH_AUTH_SOCK manually, but not from the vscode git function/plugin

any guess?

update: if I add export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1) to my .profile and relaunch vscode, it reads it and git clone works fine, but I would like to omit this step, as it should work without it.

[templela]

I'd also like to throw my 2c in the mix. Sometimes when the window is reloaded, or some other state change happens the SSH_AUTH_SOCK points at nothing. Also on every reload a new sock file is created at /tmp/vscode-ssh-auth-*.sock, this seems to be to mitigate the issues with the socket dsiappearing, but once the connection is gone none of the sockets are valid. Sometimes the variable points to a missing socket, sometimes the socket is invalid.

The symptom is similar to other people, not having a proper agent available.

For context the local machine is an M1 Pro mac on 12.5.1, the host is a Ubuntu 20.04.4 LTS machine. This is using both Remote - SSH and Dev Containers. The issue doesn't happen when only using Remote - SSH on the exact same host.