Occasional install to disk fails - BLKRRPART, usually requires reboot, re-do install steps to proceed

[jl-montes]

We've seen for the past several months occasional and random behavior when attempting disk installations after PXE booting to an in-memory version of CoreOS.

What we notice is that the coreos-install will fail and the last error indicates BLKRRPART: Device or resource busy

The work-around we've typically employed it to reboot, PXE again to CoreOS in-memory, then attempt the disk install again, 98-99% of the time we never see the error again and we get a Successful install to disk

Attached is a sample screen of when the random failure happens.

We've seen this on bare-metal blade servers and pizza-box servers, KVM vm's, and Hyper-V vm;s in the past.

[marineam]

Hm, it is possible that we are racing with udev. I don't know for sure but maybe if udev triggered BLKRRPART before us and is now probing the filesystems on the partitions the open partition device node(s) will cause the disk do be considered in-use. We could call udevadm settle but due to the way udev detects changes to disks it is hard to avoid racing with it. I will need to do more research to figure out the best solution.

[jumanjiman]

fyi: we've seen the same issue and employeed same workaround as @jl-montes

[wdennis]

Same problem experienced tonight (v444.4.0) on a bare-metal install from CD; booted again & retried install, happened again. Funny thing is after that I booted from disk to see what would happen, CoreOS was actually installed, just didn't pick up my cloud-init (which was on a path mounted from USB drive.)

[ChrisGuilbault]

I ran into this same issue repeated. Posted to the google group, but I ran into this issue 19 out of 20 times even with rebooting.

However I found a second work around that does not require a reboot.

After receiving the BLKRRPART error then I unmounted the drive I was installing to fixed the GPT to span the entire drive and ran the install script again. This worked 100% of the time. I tested it on three machines that I received the BLKRRPART or segfault errors on while trying to install to disk.

[cimnine]

Same for me as for @wdennis: Script failed with BLKRRPART, but coreos was actually installed and it booted. To be sure everything is alright I shutdown the instance of CoreOS that got installed, re-ran the ISOLINUX and re-ran the install script once again. This time: Success!

Some more info that might be helpfull: I have an Intel NUC. The disc is some SSD. The disk had a multi-partition layout before. (Previously installed was XMBCbuntu with 'default' XMBCbuntu partition layout. Not that I expect XMBCbuntu to be the cause of the trouble, but it might help you to get an idea of the previously configured partition layout.)

[apscomp]

this error also happens when installing via ISO (CD-ROM), however, if you actually eject the CD, and reboot... you will find that it has indeed installed coreOS....

[domq]

It seems unlikely that udev is the culprit: I just tried killing it until systemd gave up restarting it, and still

bash-4.2# sfdisk -R /dev/sda
sfdisk: BLKRRPART: Device or resource busy
sfdisk: This disk is currently in use.

Using CoreOS stable (CoreOS 607.0.0), also PXE-booted. I am at a loss to determine what is actually keeping the partitions busy, but since I can reproduce this at will, please feel free to suggest commands to try.

[marineam]

Interesting, so there must be two races going on. There is a race with udev, it uses inotify to detect if the full-disk node has been written to and will then issue the same ioctl to have the kernel reprobe the device and then as it receives the uevent back from the kernel it will recreate /dev/disk/by-* symlinks so if you mount by label or similar too quickly it can fail due to the missing link. We observe this while building CoreOS images from time to time.

The explicit reprobe in this script is there because only relatively recent versions of udev have this automatic behavior and even in versions that do there isn't a reliable way to wait for udev to do its magic. I previously assumed that the busy errors were due to our reprobe happening while udev was working but if that is not the case we have a whole new mystery on our hands. On Mar 19, 2015 11:33 AM, "domq" [email protected] wrote:

It seems unlikely that udev is the culprit: I just tried killing it until systemd gave up restarting it, and still

bash-4.2# sfdisk -R /dev/sda sfdisk: BLKRRPART: Device or resource busy sfdisk: This disk is currently in use.

— Reply to this email directly or view it on GitHub https://github.com/coreos/bugs/issues/152#issuecomment-83706186.

[cybertk]

encounter this issue on 633.1.0, any update?

[threedliteguy]

I had the same error installing from ISO. From USB I got Input/Output Error. Pulling up gparted from the Ubuntu live cd, it fixed errors in the GPT layout, which was all small partitions (I was installing to a 1TB HD). Reran twice, same error.

[scari]

Same error here. PXE booted and Installing from stable channel. Just followed same workaround as @jl-montes but the success rate is way lower.

[tdeheurles]

Same issue installing with 745.1.0.

I had the issue installing from a ubuntu key, and then work.

Now I try to reinstall from inside the running coreos. Can someone confirm that the command coreos-install can be executed from the running os ?

[stresler]

We ran into this several times, and I think I found the culprit in at least one scenario.

We kept encountering this with client provisions and couldn't reproduce in lab until we used the client's exact userdata, and then it was intermittent.

When cloud-init contains anything that involves a download it seems to lock the device. The weird part is it does this even if the disk doesn't have a filesystem on it at all.

I suspect docker or something else in third-party expects a disk to be present and if it isn't it picks what it thinks should be the first one /dev/sda and tries to use it.

For our installer image, I removed any access to userdata until the second boot and haven't encountered it again yet. Will update if I see it again.

[domq]

So in the case of our farm of bare-metal boxes, I think that the issue was pre-existing LVM volumes .

Zapping them with vgremove prior to running coreos-install solves the issue for me. Teaching coreos-install to do same could be worthwhile, although slightly trickier for multi-disk systems.

[blemmenes]

I can confirm using vgremove as @domq mentioned just worked for me on baremetal booted via IPMI mounted ISO.

[Evidlo]

I had this issue, too. The disk used to be part of a RAID.

I went ahead and reformatted the disk with fdisk and it installed successfully. Not sure if this is actually the cause of the success though.

[Wintereise]

Just ran into this, device used to be part of a LVM volume group.

Will remove metadata manually and see if it works.

[wgzhao]

I had this issue, too. I use system rescuecd liveOS booting,then ran core-install /dev/sda .but it give me BLKRRPART: Device or resource busy I comment out blockdev --rereadpt "${DEVICE}

at about line 303

It's works.

[stresler]

It seems like this issue is getting a bit muddied. Unfortunately the BLKRRPART error can be caused by a variety of reasons and sometimes is legitimate reporting (device is actually busy).

I think for this issue to remain valid we need to create specific reproduction steps and treat each verified set of reproduction steps as a new issue. The issue we had has been resolved (see this comment).

Does anyone have specific reproduction steps? Otherwise I suggest archiving this for reference and treating new instances as new issues with a focus on tracking it down per hardware setup.

Does that sound reasonable?

[gbock]

One case is definitely active LVM Volume groups on DEVICE from a previous install of another OS (my use case is having to install CoreOS via grub from a previous CentOS install).

Pre Install state:

localhost ~ # parted -s /dev/sda print
Model: ATA CentOS to CoreOS (scsi)
Disk /dev/sda: 68.7GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  525MB   524MB   primary  ext4         boot
 2      525MB   68.7GB  68.2GB  primary               lvm

localhost ~ # lsblk 
NAME                 MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                    8:0    0    64G  0 disk 
|-sda1                 8:1    0   500M  0 part 
`-sda2                 8:2    0  63.5G  0 part 
  |-VolGroup-lv_home 254:0    0    31G  0 lvm  
  |-VolGroup-lv_root 254:1    0  30.5G  0 lvm  
  `-VolGroup-lv_swap 254:2    0     2G  0 lvm  
sr0                   11:0    1  96.4M  0 rom  
sr1                   11:1    1   395M  0 rom  
loop0                  7:0    0 182.1M  0 loop /usr

Post Install state:

localhost ~ # parted -s /dev/sda print
Warning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 124928000 blocks) or continue with the current setting? 
Model: ATA CentOS to CoreOS (scsi)
Disk /dev/sda: 68.7GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: pmbr_boot

Number  Start   End     Size    File system  Name        Flags
 1      2097kB  136MB   134MB   fat16        EFI-SYSTEM  boot, legacy_boot, esp
 2      136MB   138MB   2097kB               BIOS-BOOT   bios_grub
 3      138MB   1212MB  1074MB  ext2         USR-A
 4      1212MB  2286MB  1074MB               USR-B
 6      2286MB  2420MB  134MB   ext4         OEM
 7      2420MB  2487MB  67.1MB               OEM-CONFIG
 9      2487MB  4754MB  2267MB  ext4         ROOT

localhost ~ # lsblk 
NAME                 MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                    8:0    0    64G  0 disk 
|-sda1                 8:1    0   500M  0 part 
`-sda2                 8:2    0  63.5G  0 part 
  |-VolGroup-lv_home 254:0    0    31G  0 lvm  
  |-VolGroup-lv_root 254:1    0  30.5G  0 lvm  
  `-VolGroup-lv_swap 254:2    0     2G  0 lvm  
sr0                   11:0    1  96.4M  0 rom  
sr1                   11:1    1   395M  0 rom  
loop0                  7:0    0 182.1M  0 loop /usr

These should be set inactive before the image is written to DEVICE:

---- /usr/bin/coreos-install    2015-12-01 02:01:29.000000000 +0000
+++ /tmp/coreos-install 2015-12-03 16:20:37.535278241 +0000
@@ -284,6 +284,9 @@
 echo "Downloading the signature for ${IMAGE_URL}..."
 wget --inet4-only --no-verbose -O "${WORKDIR}/${SIG_NAME}" "${SIG_URL}"

+# Deactivate any LVM volume groups on DEVICE
+vgs -a -o +devices | awk "/ ${DEVICE//\//\/}[0-9]/{print \$1}" | sort -u | xargs -n1 vgchange -an
+
 echo "Downloading, writing and verifying ${IMAGE_NAME}..."
 declare -a EEND
 if ! wget --inet4-only --no-verbose -O - "${IMAGE_URL}" \

I'm using this unit for now in my oem cloud config (a bit more of a hammer):

coreos:
  units:
    - name: deactivate-lvm.service
      command: start
      content: |
        [Service]
        Type=oneshot
        ExecStart=/bin/sh -c '/usr/sbin/vgs --noheadings -o vg_name | /usr/bin/xargs -n1 /usr/sbin/vgchange -an'

[coconutpilot]

I can reproduce this quite easily PXE booting on a test cluster of Dell Optiplex 960 towers (desktop class machines) and running coreos-install. I wonder how this lower spec of hardware comes into play ... (slow disks, unsafe caching, etc).

This is an example of a failed install:

core@fox3 ~ $ sudo coreos-install -C alpha -V current -d /dev/sda -c fox3.yaml 
Checking availability of "local-file"
Fetching user-data from datasource of type "local-file"
Downloading the signature for http://alpha.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2...
2015-12-07 23:15:29 URL:http://alpha.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.sig [543/543] -> "/tmp/coreos-install.RsziI7s9km/coreos_production_image.bin.bz2.sig" [1]
Downloading, writing and verifying coreos_production_image.bin.bz2...
2015-12-07 23:16:13 URL:http://alpha.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2 [231125324/231125324] -> "-" [1]
gpg: Signature made Thu Dec  3 02:51:35 2015 UTC using RSA key ID 1CB5FA26
gpg: key 93D2DCB4 marked as ultimately trusted
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: Good signature from "CoreOS Buildbot (Offical Builds) <[email protected]>" [ultimate]
blockdev: ioctl error on BLKRRPART: Device or resource busy

Both sync and blockdev --flushbufs work around this problem (for me at least).

$ diff -u /usr/bin/coreos-install /tmp/coreos-install 
--- /usr/bin/coreos-install     2015-11-05 02:11:46.000000000 +0000
+++ /tmp/coreos-install 2015-12-07 23:11:29.251420878 +0000
@@ -300,7 +300,7 @@
 fi

 # inform the OS of partition table changes
-blockdev --rereadpt "${DEVICE}"
+blockdev --flushbufs --rereadpt "${DEVICE}"

 if [[ -n "${CLOUDINIT}" ]] || [[ -n "${COPY_NET}" ]]; then
     # The ROOT partition should be #9 but make no assumptions here!

[crawford]

@coconutpilot neat! Would you mind filing a PR against https://github.com/coreos/init/blob/master/bin/coreos-install?

[jumanjiman]

@coconutpilot neat! Would you mind filing a PR against

:+1:

[coconutpilot]

Before I submitted my pull request I wanted to do some more testing. A simpler test case of the bug is (may need to run a few times):

blockdev --rereadpt /dev/sda & blockdev --rereadpt /dev/sda & wait

Looking at the kernel ioctl: http://lxr.free-electrons.com/source/block/ioctl.c#L184

its wrapped in a mutex so I am at a loss as to why this is happening?

As noted by @marineam in the first comment to the issue report it seems to be a race with udev, this is where udev does BLKRRPART:

https://github.com/systemd/systemd/blob/564c44436cf64adc7a9eff8c17f386899194a893/src/udev/udevd.c#L1043

This means my proposed fix blockdev --flushbufs only worked because it gave enough time for the rescan called by udevd to complete. Additionally @marineam had it figured out from the start.

A proposal for a solution:

• add a pre-flight check that the install device has no mounted partitions
• remove the call to blockdev --rereadpt as udevd already effectively makes the same ioctl.
• add a sleep 1 between writing the filesystem to /dev/sdX and mounting. Perhaps udevadm settle could be used here, but docs focus on using that command at boot time.

Does this sound sane?

[danilochilene]

I got the same issue.

Only worked after deleting the VG.

[jotasixto]

When I tried install on disk the CoreOS stable (835.12.0) on XEN 3.16 PV Guest I had the BLKRRPAR error. My mistake was that I had the LVM volume with malformed name. I was adding a number to the name to volume label, I think that this create a conflict interpreting the volume as a partition. I rename the volume label without numbers at the end and it works fine.

[thereallukl]

I just hit it when I am installing CoreOS on machines that had CentOS installed and for me removing DM mapping prior to running installer fixes the issue:

dmsetup remove centos_cnlvr01r07s2-root
dmsetup remove centos_cnlvr01r07s2-home
dmsetup remove centos_cnlvr01r07s2-swap

centos_blah_blah names can be listed from /dev/mapper/*.

After that I can write CoreOS to /dev/sda

[ivarec]

Just bumped into this issue on a machine that was previously a RAID-1 setup. Got it to work after thrashing both disks' MBR and rebooting:

BEWARE: this will trash all data in those disks!

dd if=/dev/zero of=/dev/sda bs=512 count=1
dd if=/dev/zero of=/dev/sdb bs=512 count=1
reboot

[crawford]

@haolez which version of CoreOS was being used to run coreos_install?

[ivarec]

@crawford 1122.2.0

EDIT Ah, sorry! I was using 1122.2.0 to run coreos-install. The one mentioned previously was what I was trying to install locally.

(Not a production environment)