SSH Agent Forwarding not working during provisioning

[jansepke]

Similar to Issue GH-1303 I have problems using forward_agent during provisioning. It is working fine after privisioning. I tried different workarounds but none is working.

Vagrant version

Vagrant 2.2.13

Host operating system

Windows 10 Enterprise 1909

Guest operating system

Ubuntu (hashicorp/bionic64)

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "hashicorp/bionic64"
  config.vagrant.plugins = ["vagrant-vbguest"]
  config.ssh.forward_agent = true
  config.vm.provision "shell", path: "provision.sh", privileged: false
end

provision.sh

echo SSH_AUTH_SOCK: $SSH_AUTH_SOCK
echo ssh-agent:
ssh-agent
echo "ps:"
ps x | grep ssh-agent
echo ssh-add:
ssh-add -l

Vagrant up output

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'hashicorp/bionic64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'hashicorp/bionic64' version '1.0.282' is up to date...
==> default: Setting the name of the VM: vagrant_default_1605856315994_74479
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection reset. Retrying...
    default: Warning: Connection aborted. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: The guest additions on this VM do not match the installed version of
    default: VirtualBox! In most cases this is fine, but in rare cases it can
    default: prevent things such as shared folders from working properly. If you see
    default: shared folder errors, please make sure the guest additions within the
    default: virtual machine match the version of VirtualBox you have installed on
    default: your host and reload your VM.
    default:
    default: Guest Additions Version: 6.0.10
    default: VirtualBox Version: 6.1
==> default: Mounting shared folders...
    default: /vagrant => C:/Users/XXXXX/projects/vagrant
==> default: Running provisioner: shell...
    default: Running: C:/Users/XXXXXX/AppData/Local/Temp/vagrant-shell20201120-19364-4xunc.sh
    default: SSH_AUTH_SOCK: /tmp/ssh-mknmloXmln/agent.1246
    default: ssh-agent:
    default: SSH_AUTH_SOCK=/tmp/ssh-jshoNC894XUT/agent.1417; export SSH_AUTH_SOCK;
    default: SSH_AGENT_PID=1418; export SSH_AGENT_PID;
    default: echo Agent pid 1418;
    default: ps:
    default:  1418 ?        Ss     0:00 ssh-agent
    default:  1420 ?        S      0:00 grep ssh-agent
    default: ssh-add:
    default: error fetching identities: communication with agent failed

Output in vagrant ssh

vagrant@vagrant:~$ ssh-add -l
4096 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX C:\Users\XXXXXX\.ssh\id_rsa (RSA)

Expected behavior

ssh-add -l should output the same during provisioning and vagrant ssh.

Actual behavior

ssh agent is not forwarded during provisioning.

References

• GH-1303

[LastDragon-ru]

Vagrant 2.2.10/2.2.14 + VirtualBox 6.1.16/6.1.18 + Windows 10.0.19041 + ubuntu/focal64 + OpenSSH_for_Windows_8.1p1, LibreSSL 2.6.5 = same error :( (the agent doesn't work only while provisioning)

[cussack]

👍 same as @LastDragon-ru

[countzero]

I can also reproduce this bug with vagrant 2.3.1 and OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2 using the following minimal test case:

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "debian/bullseye64"
  config.ssh.forward_agent = true
  config.vm.provision "shell", inline: "ssh-add -L"
end

Test

1. ssh-add
2. vagrant up
3. vagrant ssh -- "ssh-add -L"

Output of inline provisioning

 default: Running: inline script
 default: error fetching identities: communication with agent failed

Output of vagrant ssh

ssh-ed25519 [...]

[countzero]

If I am running vagrant up --debug there is a line indicating, that Vagrant tries to find a pageant process.

E, [2022-10-15T21:02:06.088907 #32892] ERROR -- net.ssh.authentication.agent[26548]: 
could not connect to ssh-agent: pageant process not running

@mitchellh: Is it possible, that Vagrant uses a different SSH implementation for connections and provisioning? Is it a huge effort to also use the SSH implementation that works on vagrant ssh for the shell provisioning?

It would be really nice to support SSH agent forwarding in all contexts!

[countzero]

OK, I dug around and vagrant indeed uses https://github.com/net-ssh/net-ssh in the shell provisioning context and the available ssh binary in the vagrant ssh context. See https://github.com/hashicorp/vagrant/issues/10195 for details.

The problem is, that the net-ssh library does not yet support OpenSSH on windows: https://github.com/net-ssh/net-ssh/issues/628

@chrisroberts & @mitchellh: Is it possible to simply use the available ssh binary of the host system for the shell provisioning?

[countzero]

Until the vagrant shell provisioning is fully cross-platform compatible, we can use the following workaround:

Vagrantfile

ENV['VAGRANT_EXPERIMENTAL'] = "typed_triggers"

Vagrant.configure("2") do |config|
  config.ssh.forward_agent = true
  config.vm.box = "debian/bullseye64"

  config.vm.provision "shell", inline: 'echo "net-ssh library based provisioning"'

  config.trigger.after :provisioner_run, type: :hook do |trigger|
    trigger.info = "ssh binary based provisioning that is cross-platform compatible"
    trigger.run = {
      inline: 'vagrant ssh -- "ssh-add -L"'
    }
  end
end

This utilizes the experimental https://www.vagrantup.com/docs/triggers/configuration#trigger-types hook to remotely execute anything via the vagrant ssh command, which uses the ssh binary of the host machine.

It is a bit ugly, because we need at least one regular provisioning task in the Vagrantfile. But otherwise it works as expected: The trigger will only be executed in a provisioning context.