Updating the Vagrant box breaks SSH key authentication

[ottok]

If one runs vagrant up for an existing project after vagrant box update was run, one will not be able to access the box anymore due to mismatch in SSH keys:

$ vagrant up
Bringing machine 'wordpress-box' up with 'virtualbox' provider...
==> wordpress-box: Checking if box 'seravo/wordpress' is up to date...
==> wordpress-box: Clearing any previously set forwarded ports...
==> wordpress-box: Clearing any previously set network interfaces...
==> wordpress-box: Preparing network interfaces based on configuration...
    wordpress-box: Adapter 1: nat
    wordpress-box: Adapter 2: hostonly
==> wordpress-box: Forwarding ports...
    wordpress-box: 22 (guest) => 2222 (host) (adapter 1)
==> wordpress-box: Running 'pre-boot' VM customizations...
==> wordpress-box: Booting VM...
==> wordpress-box: Waiting for machine to boot. This may take a few minutes...
    wordpress-box: SSH address: 127.0.0.1:2222
    wordpress-box: SSH username: vagrant
    wordpress-box: SSH auth method: private key
    wordpress-box: Warning: Authentication failure. Retrying...
    wordpress-box: Warning: Authentication failure. Retrying...
    wordpress-box: Warning: Authentication failure. Retrying...
    wordpress-box: Warning: Authentication failure. Retrying...
    wordpress-box: Warning: Authentication failure. Retrying...
    wordpress-box: Warning: Authentication failure. Retrying...
    wordpress-box: Warning: Authentication failure. Retrying...

The workaround is to run vagrant destroy; vagrant up but it seems stupid. Investigate how and where the SSH key is generated, and how we could save it and reuse if permanently to avoid this situation in the future.

[tuanalumi]

I second this issue. Happened to me with ubuntu/xenial64 box.

[ottok]

This is still the case with our latest image:

$ vagrant up
Bringing machine 'redacted-box' up with 'virtualbox' provider...
==> redacted-box: Checking if box 'seravo/wordpress-beta' is up to date...
==> redacted-box: Clearing any previously set forwarded ports...
==> redacted-box: Clearing any previously set network interfaces...
==> redacted-box: Preparing network interfaces based on configuration...
    redacted-box: Adapter 1: nat
    redacted-box: Adapter 2: hostonly
==> redacted-box: Forwarding ports...
    redacted-box: 22 (guest) => 2222 (host) (adapter 1)
==> redacted-box: Running 'pre-boot' VM customizations...
==> redacted-box: Booting VM...
==> redacted-box: Waiting for machine to boot. This may take a few minutes...
    redacted-box: SSH address: 127.0.0.1:2222
    redacted-box: SSH username: vagrant
    redacted-box: SSH auth method: private key
    redacted-box: Warning: Authentication failure. Retrying...
    redacted-box: Warning: Authentication failure. Retrying...
    redacted-box: Warning: Authentication failure. Retrying...
    redacted-box: Warning: Authentication failure. Retrying...
    redacted-box: Warning: Authentication failure. Retrying...
    redacted-box: Warning: Authentication failure. Retrying...
    redacted-box: Warning: Authentication failure. Retrying...
    redacted-box: Warning: Authentication failure. Retrying...
    redacted-box: Warning: Authentication failure. Retrying...

Users need to vagrant destroy && vagrant up after running vagrant box update.

I guess it is a "feature" of Vagrant. Can we try to make some sort of workaround?

[ypcs]

We can just toggle off the feature if we want to go the easy way (config.ssh.insert_key = false). However, this decreases security as then all environments share known keys. (Which was exactly the reason for upstream implementing this feature.)

Issue seems to be that when Vagrant first starts up, well-known public key gets replaced with instance-specific key. Private part of this key get stored to .vagrant. When box gets updated, generated (public) key gets wiped from the box (but not from host), host tries to use existing key & this causes the error above.

Hypothesis above should/could be confirmed by examining vagrant ssh-config before/after box update.

[ottok]

This is a known issue inherited from upstream. Vagrant by default always uses the same keys globally, which is very insecure, and if keys are recreated per box then there is no reliable method to retain them across box updates. Also, in some cases network issues or others will result in the SSH step being stuck.

See https://github.com/hashicorp/vagrant/issues/8157

Verbose explanation of this upstream feature in Finnish:

Taustaa: Vagrant-boxit toimivat siten, että boxin sisällä on yleisesti tiedossa oleva SSH-avain, jota käytetään kun boxiin kirjaudutaan sisään. Tämä pitää sisällään tietoturva-ongelman: avain on yleisesti tiedossa, ja vapaasti saatavilla verkossa. Tästä syystä Vagrant on muutaman vuoden ajan toiminut siten, että kun boxia käynnistetään ensimmäistä kertaa, tuo boxin sisällä oleva SSH-avain korvataan uudella projektikohtaisella avaimella, joka tallennetaan .vagrant -hakemiston alle projektikansioon. Seuraavilla kirjautumiskerroilla Vagrant käyttää tätä uutta avainta tunnistautumiseen ja boxiin kirjautumiseen.

Nyt kun käyttäjä päivittää boxin uudempaan versioon, tuo paikallisesti generoitu avain jää paikalliseen tiedostojärjestelmään, mutta "uudelleenasennetusta" boxista sitä ei löydykään, vaan tilalla on taas se yleisesti tiedossa oleva turvaton oletusavain. Vagrant kuitenkin yrittää kirjautumista generoidulla avaimella, mikä ei tietenkään toimi. Issuessa mukana olevan vagrant ssh-config -listauksen perusteella vagrant yrittää käyttää avainta polusta IdentityFile /Users/xyz/.vagrant.d/boxes/seravo-VAGRANTSLASH-wordpress/20180605.0.4/virtualbox/vagrant_private_key.

Ensiapuna voisi yrittää siirtää tuon hakemiston talteen & tämän jälkeen suorittaa vagrant up uudelleen: mv .vagrant .vagrant.old && vagrant up. Tällöin järjestelmä yrittää taas käyttää tuota oletusavainta + ei löydä generoitua avainta järjestelmästä.

[zArubaru]

As a decent quick fix, the following should work. Copy the old vagrant_private_key to the new box:

cd ~/.vagrant.d/boxes/seravo-VAGRANTSLASH-wordpress/20180605.0.4/virtualbox
cp ../../20170829.14.0000/virtualbox/vagrant_private_key vagrant_private_key.old

Then edit Vagrantfile in the same directory: Line: config.ssh.private_key_path = File.expand_path("../vagrant_private_key", __FILE__) to config.ssh.private_key_path = [File.expand_path("../vagrant_private_key", __FILE__), File.expand_path("../vagrant_private_key.old", __FILE__)]

After this new and old projects should work painlessly.

[ottok]

So apparently config.ssh.private_key_path can be an array with multiple keys. We need to test if we can pimp https://github.com/Seravo/wordpress/blob/master/Vagrantfile to always use as many available keys as possible to ensure the connection to the old and new development Vagrant boxes "just works" and always.

..however I haven't been able to reproduce this issue for a long time, so it is a bit difficult to test if a fix works or not since there is no known test case i.e. way to reproduce this bug.

[zArubaru]

@ottok

So apparently config.ssh.private_key_path can be an array with multiple keys.

If you read my previous comment two years ago, this would've been apparent, well, two years ago :D

Glad to hear this doesn't seem to be causing issues anymore, I can't really say as I haven't been using this for a while now.

[ottok]

The issue was not assigned to me 2 years ago, so I did not follow up it in detail then. I apologize this didn't get proper attention back then.

[ottok]

I have not been able to reproduce this for a while, so I don't have a case where to test potential fixes.

[JoosuaKoskinen]

WIP fix: https://github.com/Seravo/wordpress/commit/0ffdb1e43cf37632473dc55fc0674377e6f5e170