[macOS] Multipass loses mounts on restart

[akauppi]

Edit

• Look for the comments instead of this original Issue description.
• TL;DR multipass restart after umount of a classic mount causes instability in Multipass 1.14.0 (on macOS; at least Intel)
• Work-around: instead of multipass restart, use separate multipass stop and multipass start commands.

Original problem description follows

Describe the bug

I updated the Ubuntu stack today and immediately restarted the VM (because there was a new kernel the apt output suggests this). During the restart I get an error message.

This makes me think perhaps the kernel update and the losing of the mounts are related.

To Reproduce

1. It is a bit difficult to reproduce this, since one would need an upstream kernel update to be available. Once that is the case:

   $ sudo apt update; sudo apt upgrade
   [...]
   
   Pending kernel upgrade!
   Running kernel version:
     6.8.0-40-generic
   Diagnostics:
     The currently running kernel version is not the expected kernel version 6.8.0-41-generic.
   
   Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting.
   
   No services need to be restarted.
   
   No containers need to be restarted.
   
   No user sessions are running outdated binaries.
   
   No VM guests are running outdated hypervisor (qemu) binaries on this host.
   

2. Exit from VMs and:

   $ mp restart rust-emb
   Removing mount "/home/ubuntu/Rustyard" from 'rust-emb': [ssh proc] failed to open session channel: 'Socket error: Connection reset by peer'
   Removing mount "/home/ubuntu/VL53L5CX_rs" from 'rust-emb': ssh connection failed: 'Socket error: Connection reset by peer'
   

Expected behavior

I would like the mounts to have remained.

Logs

Please provide logs from the daemon, see accessing logs on where to find them on your platform.

Logs (no errors)

[2024-08-21T09:02:54.724] [debug] [async task] fetch manifest periodically
[2024-08-21T09:03:04.722] [warning] [url downloader] Error getting https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Network timeout - trying cache.
[2024-08-21T09:03:04.722] [warning] [url downloader] Cannot retrieve headers for https://cdimage.ubuntu.com/ubuntu-core/18/stable/current/ubuntu-core-18-amd64.img.xz: Network timeout
[2024-08-21T09:03:04.722] [warning] [url downloader] Error getting https://cloud-images.ubuntu.com/buildd/daily/streams/v1/index.json: Network timeout - trying cache.
[2024-08-21T09:03:04.722] [warning] [url downloader] Cannot retrieve headers for https://cdimage.ubuntu.com/ubuntu-core/20/stable/current/ubuntu-core-20-amd64.img.xz: Network timeout
[2024-08-21T09:03:04.722] [warning] [url downloader] Cannot retrieve headers for https://cdimage.ubuntu.com/ubuntu-core/16/stable/current/ubuntu-core-16-amd64.img.xz: Network timeout
[2024-08-21T09:03:04.723] [warning] [url downloader] Cannot retrieve headers for https://cdimage.ubuntu.com/ubuntu-core/22/stable/current/ubuntu-core-22-amd64.img.xz: Network timeout
[2024-08-21T09:03:04.723] [warning] [VMImageHost] Could not update manifest: failed to download from 'https://cdimage.ubuntu.com/ubuntu-core/16/stable/current/ubuntu-core-16-amd64.img.xz': Operation canceled
[2024-08-21T09:03:14.723] [warning] [url downloader] Error getting https://cloud-images.ubuntu.com/buildd/daily/streams/v1/com.ubuntu.cloud:daily:download.json: Network timeout - trying cache.
[2024-08-21T09:08:43.412] [debug] [rust-emb] QMP: {"timestamp": {"seconds": 1724220523, "microseconds": 412178}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[5]/rtc"}}

[2024-08-21T09:14:45.931] [debug] [ssh session] Executing 'findmnt --type 9p | grep '/home/ubuntu/Rustyard m9586b90902ea3195b09af3dd6e5645''
[2024-08-21T09:14:47.007] [debug] [ssh session] Executing 'findmnt --type 9p | grep '/home/ubuntu/Embassy-playground-2024 me67e8b88cd6831cd9f2edd9777d03d''
[2024-08-21T09:14:47.130] [info] [qemu-mount-handler] Stopping native mount "/home/ubuntu/Embassy-playground-2024" in instance 'rust-emb'
[2024-08-21T09:14:47.159] [debug] [ssh session] Executing 'if mountpoint -q /home/ubuntu/Embassy-playground-2024; then sudo umount /home/ubuntu/Embassy-playground-2024; else true; fi'
[2024-08-21T09:14:47.406] [debug] [ssh session] Executing 'findmnt --type 9p | grep '/home/ubuntu/VL53L5CX_rs m10ff1dc258763069a8c3c573cf2864''
[2024-08-21T09:14:47.528] [info] [qemu-mount-handler] Stopping native mount "/home/ubuntu/VL53L5CX_rs" in instance 'rust-emb'
[2024-08-21T09:14:47.554] [debug] [ssh session] Executing 'if mountpoint -q /home/ubuntu/VL53L5CX_rs; then sudo umount /home/ubuntu/VL53L5CX_rs; else true; fi'
[2024-08-21T09:14:47.681] [debug] [daemon] Rebooting rust-emb
[2024-08-21T09:14:47.681] [debug] [ssh session] Executing 'sudo systemctl stop ssh'
[2024-08-21T09:14:47.718] [debug] [ssh session] Executing 'sudo reboot'
[2024-08-21T09:14:47.773] [debug] [rust-emb] Dropping cached SSH session
[2024-08-21T09:14:47.773] [debug] [rust-emb] Waiting for SSH to be up
[2024-08-21T09:14:47.876] [debug] [rust-emb] Caching initial SSH session
[2024-08-21T09:14:47.927] [debug] [ssh session] Executing 'sudo /bin/bash -c 'P="/home/ubuntu/Rustyard"; while [ ! -d "$P/" ]; do P="${P%/*}"; done; echo $P/''
[2024-08-21T09:14:48.032] [warning] [daemon] Removing mount "/home/ubuntu/Rustyard" from 'rust-emb': [ssh proc] failed to open session channel: 'Socket error: Connection reset by peer'

[2024-08-21T09:14:48.527] [debug] [ssh session] Executing 'sudo /bin/bash -c 'P="/home/ubuntu/Embassy-playground-2024"; while [ ! -d "$P/" ]; do P="${P%/*}"; done; echo $P/''
[2024-08-21T09:14:48.759] [debug] [ssh session] Executing 'sudo mount -t 9p me67e8b88cd6831cd9f2edd9777d03d /home/ubuntu/Embassy-playground-2024 -o trans=virtio,version=9p2000.L,msize=536870912'
[2024-08-21T09:14:48.784] [warning] [daemon] Removing mount "/home/ubuntu/VL53L5CX_rs" from 'rust-emb': ssh connection failed: 'Socket error: Connection reset by peer'

[2024-08-21T09:14:52.702] [debug] [rust-emb] QMP: {"timestamp": {"seconds": 1724220892, "microseconds": 701567}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}}

[2024-08-21T09:14:52.702] [info] [rust-emb] VM restarting
[2024-08-21T09:14:52.703] [debug] [rust-emb] Dropping cached SSH session
[2024-08-21T09:15:11.737] [debug] [ssh session] Executing 'findmnt --type 9p | grep '/home/ubuntu/Embassy-playground-2024 me67e8b88cd6831cd9f2edd9777d03d''
[2024-08-21T09:15:12.133] [debug] [rust-emb] Waiting for SSH to be up
[2024-08-21T09:15:12.133] [debug] [rust-emb] QMP: {"timestamp": {"seconds": 1724220893, "microseconds": 219597}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[5]/rtc"}}
{"timestamp": {"seconds": 1724220893, "microseconds": 402854}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[5]/rtc"}}

[2024-08-21T09:15:12.161] [debug] [rust-emb] Caching initial SSH session
[2024-08-21T09:15:12.193] [debug] [ssh session] Executing 'sudo /bin/bash -c 'P="/home/ubuntu/Embassy-playground-2024"; while [ ! -d "$P/" ]; do P="${P%/*}"; done; echo $P/''
[2024-08-21T09:15:12.393] [debug] [ssh session] Executing 'sudo mount -t 9p me67e8b88cd6831cd9f2edd9777d03d /home/ubuntu/Embassy-playground-2024 -o trans=virtio,version=9p2000.L,msize=536870912'
[2024-08-21T09:15:59.433] [debug] [rust-emb] QMP: {"timestamp": {"seconds": 1724220959, "microseconds": 433605}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[5]/rtc"}}

[2024-08-21T09:17:54.749] [debug] [async task] fetch manifest periodically
[2024-08-21T09:18:04.748] [warning] [url downloader] Cannot retrieve headers for https://cdimage.ubuntu.com/ubuntu-core/20/stable/current/ubuntu-core-20-amd64.img.xz: Network timeout
[2024-08-21T09:18:04.748] [warning] [url downloader] Error getting https://cloud-images.ubuntu.com/buildd/daily/streams/v1/index.json: Network timeout - trying cache.
[2024-08-21T09:18:04.748] [warning] [url downloader] Cannot retrieve headers for https://cdimage.ubuntu.com/ubuntu-core/22/stable/current/ubuntu-core-22-amd64.img.xz: Network timeout
[2024-08-21T09:18:04.748] [warning] [url downloader] Cannot retrieve headers for https://cdimage.ubuntu.com/ubuntu-core/18/stable/current/ubuntu-core-18-amd64.img.xz: Network timeout
[2024-08-21T09:18:04.749] [warning] [url downloader] Error getting https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Network timeout - trying cache.
[2024-08-21T09:18:04.749] [warning] [url downloader] Cannot retrieve headers for https://cdimage.ubuntu.com/ubuntu-core/16/stable/current/ubuntu-core-16-amd64.img.xz: Network timeout
[2024-08-21T09:18:04.750] [warning] [VMImageHost] Could not update manifest: failed to download from 'https://cdimage.ubuntu.com/ubuntu-core/16/stable/current/ubuntu-core-16-amd64.img.xz': Operation canceled
[2024-08-21T09:18:14.749] [warning] [url downloader] Error getting https://cloud-images.ubuntu.com/buildd/daily/streams/v1/com.ubuntu.cloud:daily:download.json: Network timeout - trying cache.
[2024-08-21T09:27:01.442] [debug] [rust-emb] QMP: {"timestamp": {"seconds": 1724221621, "microseconds": 442521}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[5]/rtc"}}

• OS: macOS 14.6.1 (Intel)

• multipass version

  multipass   1.14.0-rc1.2439+g95e4d16a3.mac
  multipassd  1.14.0-rc1.2439+g95e4d16a3.mac
  

• multipass info

  Name:           rust-emb
  State:          Running
  Snapshots:      0
  IPv4:           192.168.64.120
  Release:        Ubuntu 24.04 LTS
  Image hash:     d2377667ea95 (Ubuntu 24.04 LTS)
  CPU(s):         3
  Load:           0.00 0.00 0.00
  Disk usage:     5.2GiB out of 17.4GiB
  Memory usage:   402.2MiB out of 5.8GiB
  Mounts:         /Users/asko/Git/Embassy-playground-2024 => /home/ubuntu/Embassy-playground-2024
                   UID map: 501:default
                   GID map: 20:default
  

  One mount (out of three) seems to have survived. Don't know why.

• multipass get local.driver

  qemu
  

Additional context

Using a RC-1 because of issues with 1.14.0 proper.

The mounts were made using --type=native (I always do that), e.g.

$ mp mount --type=native Rustyard rust-emb:/home/ubuntu/Rustyard

Work-around:

• recreate the mounts manually

[ricab]

Hi @akauppi. Thanks for reporting. I can reproduce the issue just by restarting, so I don't think this is related to the update.

Strangely, I observe this with either QEMU + native mounts or VirtualBox + classic mounts, but not the other way around. The issue also did not manifest on Linux/QEMU.

[akauppi]

I noticed you can reproduce the problem. Great.

Here's anyways steps regarding where I face it. The script fails at the end, in a umount + restart + exec. It loses connection to the Multipass service, which seems rather disturbing to me.

There seems to be a simple work-around, though.

$ git clone http://github.com/akauppi/mp
$ cd mp
$ git checkout 2fc33fc
$ USE_NATIVE_MOUNT=0 ./rust/prep.sh 
   Starting rust -
   [...]

   Multipass IP (rust): 192.168.64.130

   exec failed: ssh connection failed: 'Socket error: Connection reset by peer'

Expected

I can use either classical mounts, or native mounts, at will.

Actual

Using classical mounts, and restarting the VM causes instability. Repeatable.

Work-around

If I replace the multipass restart with separate multipass stop and multipass start, the whole problem is gone!

This is fully doable, but obviously something needs attention within the internals of Multipass 1.14.0.

[akauppi]

As an added detail, I noticed that any use of classical mounts on my system causes instabilities later on (e.g. multipass stop not being able to stop an instance). Thus, I have now turned fully to native mounts only.

Writing this just in case someone would benefit. If you use a Mac, classical mounts and have stability problems, try with native mounts.

Multipass 1.15.0 macOS 15.2 (Intel)

[romanpolunin]

Same here, restart removes all mounts, and sshfs isn't reliable - had to switch to native mounts. Mac M4 Pro, Sequoia 15.5.