`microceph disk add` can run mkfs against unavailable and in-use disks

[nobuto-m]

microceph 19.2.0+snap9ecc402e1e 1280 squid/edge canonical✓ -

microceph allows users to run disk add action and start running mkfs without --wipe option. A typo in a command like mixing up PCI IDs (/dev/disk/by-path/pci-0000:04:00.0-nvme-2 vs /dev/disk/by-path/pci-0000:05:00.0-nvme-3) could lead to a disaster although mkfs actually failed luckily.

How to reproduce:

uvt-simplestreams-libvirt sync release=noble arch=amd64
uvt-kvm create --cpu 4 --memory 4096 foo release=noble

uvt-kvm wait foo
uv-kvm ssh foo

$ lsblk 
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0     7:0    0  44.4M  1 loop /snap/snapd/23545
loop1     7:1    0  66.2M  1 loop /snap/core24/716
loop2     7:2    0 111.5M  1 loop /snap/microceph/1280
vda     253:0    0     8G  0 disk 
├─vda1  253:1    0     7G  0 part /
├─vda14 253:14   0     4M  0 part 
├─vda15 253:15   0   106M  0 part /boot/efi
└─vda16 259:0    0   913M  0 part /boot
vdb     253:16   0   372K  0 disk 

sudo snap install microceph --channel squid/edge

sudo microceph cluster bootstrap

$ sudo microceph disk list --json
{"ConfiguredDisks":[],"AvailableDisks":[]}

As above, there is no available disks. However, the disk add commands allows to start formatting /dev/vda, which is the root volume with the OS without requesting --wipe option.

$ sudo microceph disk add /dev/vda 

+----------+---------+
|   PATH   | STATUS  |
+----------+---------+
| /dev/vda | Failure |
+----------+---------+
Error: failed to bootstrap OSD: Failed to run: ceph-osd --mkfs --no-mon-config -i 1: exit status 250 (2025-02-09T13:58:31.743+0000 7a0636cce600 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1/block at offset 4096: End of buffer [buffer:2]
2025-02-09T13:58:31.743+0000 7a0636cce600 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1/block at offset 4096: End of buffer [buffer:2]
2025-02-09T13:58:31.743+0000 7a0636cce600 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-1/block at offset 4096: End of buffer [buffer:2]
2025-02-09T13:58:31.753+0000 7a0636cce600 -1 bdev(0x55db03590000 /var/lib/ceph/osd/ceph-1/block) open open got: (16) Device or resource busy
2025-02-09T13:58:31.753+0000 7a0636cce600 -1 bluestore(/var/lib/ceph/osd/ceph-1) mkfs failed, (16) Device or resource busy
2025-02-09T13:58:31.753+0000 7a0636cce600 -1 OSD::mkfs: ObjectStore::mkfs failed with error (16) Device or resource busy
2025-02-09T13:58:31.753+0000 7a0636cce600 -1  ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1: (16) Device or resource busy)

[syncronize-issues-to-jira[bot]]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CEPH-1170.

This message was autogenerated

[nobuto-m]

What happens if I have a partitioned disk but not mounted? Does microceph destroy the data in it without --wipe option?

[UtkarshBhatthere]

Thanks for creating the issues @nobuto-m. I will push their priorities up.