Failed host key verification during bootstrap is not reported to the user

[MggMuggins]

Description

If Juju partially bootstraps a controller but can't connect to it for some reason and the user aborts the bootstrap during the "Attempting to connect to" phase, the Juju client will silently refuse to connect to a new controller bootstrapped using the same IP address (common in a MicroCloud environment).

Priority: lowish

Juju version

4.0/edge (4.0.1-6711461)

Cloud

LXD

$ snap list
...
lxd               5.21.4-9eb1368          36903  5.21/candidate   canonical✓    in-cohort
microceph         19.2.1+snap74c0060321   1582   squid/stable     canonical✓    -
microcloud        2.1.1-d49bea6           1840   2/stable         canonical✓    -
microovn          24.03.6+snap93c643f0c6  960    24.03/candidate  canonical✓    -

Expected behaviour

The Juju client should produce an error message noting that the host keys for the bootstrapped controller's IP have changed.

Optionally, the juju client should remove host keys for destroyed containers/VMs from it's known-hosts during teardown so that this doesn't occur.

Reproduce / Test

1. Spin up a Single-node MicroCloud or a LXD server with OVN.
2. Ensure that your default network is an OVN network. Prevent tenants on the OVN network from being accessed from the outside world (default ipv4.nat=true or use a firewall rule)
3. juju bootstrap localhost test-ovn and cancel the operation with Ctrl+C when the container has an IP and the "Attempting to connect to" phase has lasted a little while.
4. Configure the OVN network to be accessible from outside (ipv4.nat=false and a route on your network's router for the OVN router's external IP, 10.102.119.0/24 via 10.0.0.29 dev br-lan metric 5 in my lab)
5. juju bootstrap localhost test-ovn and observe endless "Attempting to connect to". Running the command with --debug shows the error message:

20:29:32 DEBUG juju.provider.common bootstrap.go:709 connection attempt for 10.102.119.2 failed: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:4I3xHjEeQtp8DG3Yjb7hm3x7oWewV8BIBK0kbwhy6AQ.
Please contact your system administrator.
Add correct host key in /tmp/juju-known-hosts1733470412 to get rid of this message.
Offending ED25519 key in /tmp/juju-known-hosts1733470412:3
  remove with:
  ssh-keygen -f '/tmp/juju-known-hosts1733470412' -R '10.102.119.2'
Host key for 10.102.119.2 has changed and you have requested strict checking.
Host key verification failed.

Notes & References

Workaround by recreating the OVN network.

[anvial]

We will clarify the logging, but we shall not remove host keys for destroyed containers/VMs.

[jujubot]

This bug has now been triaged. JUJU-8935