[lxd] Existing instances are not being reported immediately

[townsend2010]

Why instances silently disappear with no explanation is very hard to diagnose with Multipass on Ubuntu 22.04 LTS, where it's a very regular problem (for reasons I don't understand), certainly since Ubuntu 22.04 was released 2022-04-21.

One of the reasons it's very hard to diagnose, is that multipass list sometimes shows erroneous and misleading output making the (real) problems here appear even worse.

Sometimes the instances are permanently gone. Other times they reappear a few minutes later. (My suspicion is that these common problems might possibly be related to apt updates being applied to the host Ubuntu 22.04 x86_64 PC and its reboot soon after; but that's just a guess and it might be false.)

Here's one example. Prior to the x86_64 PC's reboot, multipass list properly shows the instance as stopped:

# multipass list
Name                    State             IPv4             Image
primary                 Stopped           --               Ubuntu 20.04 LTS

Then after reboot, multipass list shows erroneous and misleading output:

# multipass list
No instances found.

Assuming everything was lost, I ran multipass launch 22.04 --bridge [parameters] to create another VM. But it turns out everything was not lost — this time.

After the above multipass launch ... command completed, the original instance suddenly reappeared — mysteriously starting on its own.

Commands like multipass start primary were not run, so it's additionally mysterious why Multipass started it — when that was certainly not wanted:

# multipass list
Name                    State             IPv4             Image
primary                 Running           10.88.67.121     Ubuntu 20.04 LTS
                                          [2nd IP address]
                                          [3rd IP address]
altruistic-jabiru       Running           10.88.67.40      Ubuntu 22.04 LTS

# multipass version
multipass   1.10.0-dev.522+gdae14474
multipassd  1.10.0-dev.522+gdae14474

# multipass get local.driver
lxd

# multipass get local.bridged-network
enp1s0

# uname -a
Linux ubuntu-2204-desktop 5.15.0-35-generic #36-Ubuntu SMP Sat May 21 02:24:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

This is just one of many data points. I hope it might help shed light on the internal glitches causing this.

Originally posted by @holta in https://github.com/canonical/multipass/issues/2598#issuecomment-1149056923

[townsend2010]

Hi @holta

Ah, you are using the LXD driver. So what I suspect is happening is that the lxd daemon is not ready or at least not reporting the instances yet to Multipass. Since lxd can be managed by means outside of Multipass, Multipass cannot know if the instances are truly gone via something like lxc delete primary --project=multipass or if lxd is late reporting.

I will have to do some investigation as to whether we can make this better or if there is some way to query LXD to know if it really ready or not before Multipass is truly ready to accept commands. I'm making this into a new issue since it's a problem and has nothing do with #2598.

[holta]

Thanks @townsend2010 for explaining.

Side Question, when you have time: Any idea why Multipass instances (or possibly it's just lxd instances) sometimes start on their own — when that was not wanted?

After the above multipass launch ... command completed, the original instance suddenly reappeared — mysteriously starting on its own.

Commands like multipass start primary were not run, so it's additionally mysterious why Multipass started it — when that was certainly not wanted:

[townsend2010]

Hi @holta,

Could you please provide logs and I will scan through them to see if I can glean anything from them.

[holta]

Thanks @townsend2010,

Multipass logs (14502-line output of journalctl --unit 'snap.multipass*' is 2.2 MB) are attached below:

• multipass.log

Clarifications:

1. The latest/stable version of LXD underlies this, which is currently:

   # snap list lxd
   Name  Version      Rev    Tracking       Publisher   Notes
   lxd   5.2-79c3c3b  23155  latest/stable  canonical✓  -
   

2. I reinstalled Multipass several times over the past 1.5 months, to try to get it to stabilize, as follows:

   snap remove multipass --purge
   snap remove lxd --purge
   
   snap install multipass --edge
   snap install lxd
   

[holta]

@townsend2010,

Here's another (different) failure of Multipass, that occurs quite regularly on reboot of this stock x86_64 host PC (Ubuntu 22.04 LTS with graphical desktop), if it helps to diagnose:

# multipass list
list failed: cannot connect to the multipass socket
Please ensure multipassd is running and '/var/snap/multipass/common/multipass_socket' is accessible

# systemctl status snap.multipass.multipassd.service
× snap.multipass.multipassd.service - Service for snap application multipass.multipassd
     Loaded: loaded (/etc/systemd/system/snap.multipass.multipassd.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2022-06-08 10:43:45 EDT; 16min ago
    Process: 2239 ExecStart=/usr/bin/snap run multipass.multipassd (code=exited, status=1/FAILURE)
   Main PID: 2239 (code=exited, status=1/FAILURE)
        CPU: 190ms

Jun 08 10:43:45 126-u2204-desk systemd[1]: snap.multipass.multipassd.service: Scheduled restart job, restart counter is at 6.
Jun 08 10:43:45 126-u2204-desk systemd[1]: Stopped Service for snap application multipass.multipassd.
Jun 08 10:43:45 126-u2204-desk systemd[1]: snap.multipass.multipassd.service: Start request repeated too quickly.
Jun 08 10:43:45 126-u2204-desk systemd[1]: snap.multipass.multipassd.service: Failed with result 'exit-code'.
Jun 08 10:43:45 126-u2204-desk systemd[1]: Failed to start Service for snap application multipass.multipassd.

# uptime
 11:01:37 up 18 min,  2 users,  load average: 0.00, 0.00, 0.00

Full Multipass logs (31 KB, 196 lines) from journalctl --unit 'snap.multipass*' over the past 18+ hours:

• multipass.196-more-lines.log

[townsend2010]

Hey @holta,

Ok, just quickly scanning through this log, multipassd is unable to reach the internet and then throws and unexpected exception and bails. This is a known issue and needs to be fixed. I'm failing to find the originally reported issue for this, but I'll bump the priority on getting that fixed.

[holta]

multipassd is unable to reach the internet and then throws and unexpected exception and bails

Strange as this machine has highly reliable Internet access via Ethernet.

(In any case, I'm assuming the underlying issue has to do with Ubuntu 22.04 PC's boot sequence — as this particular problem does not occur after PC boot, as far as I can tell.)

I'll bump the priority on getting that fixed.

Thank you!

[townsend2010]

I'm guessing Multipass gets started before networking is fully up and ready.

[holta]

@townsend2010 is the problem of Multipass (multipassd, etc) not starting when booting the host machine... now largely mitigated with the latest "edge" channel pre-release 1.10.0-dev.622+g47b280c7 2022-06-27 (7411) ?

Note that if so: the host machine boots way too many VM's (instances) that it should not — as these VM's (instances) were fully powered down well prior to rebooting the host machine.

(Possibly this erroneous and over-eager launching of VM's is a result of these VM's (instances) having been shut down internally using the poweroff command ??)

[townsend2010]

Hi @holta,

Are you asking if this issue should be fixed in edge version of Multipass?

(Possibly this erroneous and over-eager launching of VM's is a result of these VM's (instances) having been shut down internally using the poweroff command ??)

Yes, if you don't use multipass stop, then it's really hard for multipassd to know the state of a VM. That said, we could probably be better at detecting instance states that are modified outside of Multipass when using the LXD driver.

[holta]

Are you asking if this issue should be fixed in edge version of Multipass?

we could probably be better at detecting instance states that are modified outside of Multipass when using the LXD driver.

This would be fantastic.

There's also an (all too common!) corner case where VM's/instance's crash unexpectedly, due to power failure or internal software problems (etc) — so there really should be a Multipass/LXD option (rather like in a PC's BIOS/firmware) allowing you to toggle between either of these 2 choices:

1. Auto-boot VM's/instances after power failure / glitch / etc.
2. Do NOT auto-boot VM's/instances after power failure / glitch / etc.

[holta]

if you don't use multipass stop, then it's really hard for multipassd to know the state of a VM. That said, we could probably be better at detecting instance states that are modified outside of Multipass when using the LXD driver.

@townsend2010 are you aware that multipass stop --all is not enough to prevent instances from auto-booting the next time the host boots?

Example:

• An instance is powered off internally, using a command like poweroff
• multipass stop --all is run on the host machine.
• Host machine is rebooted, and the instance (unfortunately!) auto-boots, despite both above attempts to prevent that :/

(This should be easily fixable, no?)

PS This is separate from the question as to whether or not Multipass will one day act like a modern BIOS:

there really should be a Multipass/LXD option (rather like in a PC's BIOS/firmware) allowing you to toggle between either of these 2 choices:

1. Auto-boot VM's/instances after power failure / glitch / etc.
2. Do NOT auto-boot VM's/instances after power failure / glitch / etc.