Installing microceph reef snap cause some disks to become busy

[simondeziel]

Issue report

With the reef release (sudo snap install microceph --edge), it seems that something is now trying to automatically scan for available block devices or something similar. This caused our CI to fail forcing us to install microceph from the quincy/edge channel.

What version of MicroCeph are you using ?

$ sudo snap install microceph --edge
microceph (reef/edge) 18.2.0+snape56a71f5dd from Canonical** installed

What are the steps to reproduce this issue ?

In a CI script executing on a GitHub action runner:

1. sudo snap install microceph --edge
2. sudo swapoff /mnt/swapfile
3. sudo umount /mnt # umount the ephemeral disk of GitHub Action runner

$ sudo wipefs -a "${ephemeral_disk}"
wipefs: error: /dev/sdb: probing initialization failed: Device or resource busy
Error: Process completed with exit code 1.

https://github.com/canonical/lxd/actions/runs/7715565405/job/21030444998 shows it in action and the exact steps are https://github.com/canonical/lxd/actions/runs/7715565405/workflow#L282-L333

What happens (observed behaviour) ?

If microceph (--edge) snap is installed prior to swapoff/umount, wipefs will fail on a busy device. Switching the order to swapoff/umount/wipefs prior to installing the snap works without issue.

What were you expecting to happen ?

The available block device shouldn't become unavailable/busy after installing the microceph snap.

[simondeziel]

We've discovered #288 at the same time so maybe the 2 are connected somehow.

[simondeziel]

@tomponline hypothesized that /mnt remained mounted in the microceph's snap (see https://github.com/canonical/lxd/pull/12793#discussion_r1475158229) which would explain nicely why doing the swapoff/umount before installing the snap would side-step the issue.

[UtkarshBhatthere]

That would explain the mystery, Trent and I have had similar experiences with snap namespace. Thanks for the hint.