fedora-coreos-config
fedora-coreos-config copied to clipboard
coreos
tests/kola: adding the external kola test for luks + multipath case
Adding the external kola test for luks + multipath validation ref: https://issues.redhat.com/browse/OCPBUGS-13123 https://github.com/coreos/fedora-coreos-tracker/issues/1728
CI is hitting:
boot mounted not by UUID
We need to update the logic here to instead be something like (pseudo-code):
if we have rd.multipath=default karg; then
verify that boot.mount's What is /dev/disk/by-label/dm-mpath-boot
else if ... (same logic as before)
fi
Since the original bug was against PowerPC, can you verify that it also passes there?
Yes, I have tested it on PowerPC and it got passed.
Sep 02 09:22:37 qemu0 kola-runext-test.sh[4044]: + reboot
-- Boot b4d12cf821e64eb6a0f4eedc164611f4 --
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + . /var/opt/kola/extdata/commonlib.sh
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ IFS=' '
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ read -r -a cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1696]: ++ findmnt -nvr /sysroot -o SOURCE
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + srcdev=/dev/mapper/myluksdev
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + parent_device=/dev/mapper/mpatha
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1697]: + lsblk -pno NAME /dev/mapper/mpatha
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1698]: + grep -qw /dev/mapper/myluksdev
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + echo ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok /dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: + . /var/opt/kola/extdata/luks-test.sh
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ . /var/opt/kola/extdata/commonlib.sh
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: +++ IFS=' '
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: +++ read -r -a cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1699]: +++ findmnt -nvr /sysroot -o SOURCE
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ srcdev=/dev/mapper/myluksdev
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ [[ /dev/mapper/myluksdev == /dev/mapper/myluksdev ]]
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1700]: +++ lsblk -o TYPE /dev/mapper/myluksdev --noheadings
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ blktype=crypt
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ [[ crypt == crypt ]]
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1701]: +++ findmnt -nvr /sysroot -o FSTYPE
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ fstype=xfs
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ [[ xfs == xfs ]]
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'source is XFS on LUKS device'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'source is XFS on LUKS device'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok source is XFS on LUKS device
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1702]: +++ findmnt /sysroot -no OPTIONS
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ rootflags=ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep prjquota
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1703]: ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'root mounted with prjquota'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'root mounted with prjquota'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok root mounted with prjquota
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1704]: +++ dmsetup table myluksdev
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ table='0 20137985 crypt aes-xts-plain64 :64:logon:cryptsetup:0680a9c4-9392-4038-a76d-7c932e386fa3-d0 0 253:4 32768 2 allow_discards no_read_workqueue'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep -q allow_discards
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep -q no_read_workqueue
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'discard and custom option enabled for root LUKS'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'discard and custom option enabled for root LUKS'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok discard and custom option enabled for root LUKS
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep -q rd.multipath=default /proc/cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ expected_what=/dev/disk/by-label/dm-mpath-boot
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1708]: ++ systemctl cat boot.mount
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1709]: ++ grep -q What=/dev/disk/by-label/dm-mpath-boot
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok boot mounted by /dev/disk/by-label/dm-mpath-boot
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep root=UUID= /proc/cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1710]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/fedora-coreos-65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/vmlinuz-6.10.6-200.fc40.ppc64le rw mitigations=auto,nosmt ostree=/ostree/boot.1/fedora-coreos/65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=0680a9c4-9392-4038-a76d-7c932e386fa3=myluksdev root=UUID=9236cc89-0efd-46fe-9978-8ca6aff0f203 rw rootflags=prjquota boot=UUID=adadd9db-e857-487b-8a76-65c537af3c99
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep rd.luks.name= /proc/cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1711]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/fedora-coreos-65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/vmlinuz-6.10.6-200.fc40.ppc64le rw mitigations=auto,nosmt ostree=/ostree/boot.1/fedora-coreos/65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=0680a9c4-9392-4038-a76d-7c932e386fa3=myluksdev root=UUID=9236cc89-0efd-46fe-9978-8ca6aff0f203 rw rootflags=prjquota boot=UUID=adadd9db-e857-487b-8a76-65c537af3c99
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'found root kargs'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'found root kargs'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok found root kargs
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ grep boot=UUID= /proc/cmdline
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1712]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/fedora-coreos-65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/vmlinuz-6.10.6-200.fc40.ppc64le rw mitigations=auto,nosmt ostree=/ostree/boot.1/fedora-coreos/65304876925e9d232cf5dd7e696261f5d426a40b51c81eb51816e2e1da15295d/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=0680a9c4-9392-4038-a76d-7c932e386fa3=myluksdev root=UUID=9236cc89-0efd-46fe-9978-8ca6aff0f203 rw rootflags=prjquota boot=UUID=adadd9db-e857-487b-8a76-65c537af3c99
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ ok 'found boot karg'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ++ echo ok 'found boot karg'
Sep 02 09:23:23 qemu0 kola-runext-test.sh[1695]: ok found boot karg
Just one final sanity-check: did you test this on both RHCOS and FCOS?
I tested this only on FCOS as the RHCOS having issues in boot(found multiple parent devices for crypt device /dev/dm-0), its seems the coreos-installer fix did not landed in RHCOS.
Shell i build the RHCOS image with custom coreos-installer and try this out.?
Just one final sanity-check: did you test this on both RHCOS and FCOS?
I tested this only on FCOS as the RHCOS having issues in boot(
found multiple parent devices for crypt device /dev/dm-0), its seems the coreos-installer fix did not landed in RHCOS.
Ahh indeed. We're still on v0.21.0 there for some reason. Will inquire about that.
I just tested the RHCOS with latest build and it got booted and the external kola luks+multipath test case got passed.
[coreos-assembler]$ cosa kola run -p qemu --qemu-image ../rhcos-418.94.202412022028-0-qemu.ppc64le.qcow2 -E src/config/ 'ext.config.root-reprovision.luks.multipath'
kola run -p qemu --qemu-image ../rhcos-418.94.202412022028-0-qemu.ppc64le.qcow2 -E src/config/ ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__ Will warn on failure for kola test pattern "coreos.boot-mirror*":
__ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN ext.config.root-reprovision.luks.multipath
--- PASS: ext.config.root-reprovision.luks.multipath (170.96s)
PASS, output in tmp/kola
RHCOS kola test logs forluks+multipath:
cat tmp/kola/ext.config.root-reprovision.luks.multipath/76350551-f1ca-46d3-b0ee-adb7ab3ec484/ext.config.root-reprovision.luks.multipath.txt
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + . /var/opt/kola/extdata/commonlib.sh
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ IFS=' '
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ read -r -a cmdline
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3753]: ++ findmnt -nvr /sysroot -o SOURCE
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + srcdev=/dev/mapper/myluksdev
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + parent_device=/dev/mapper/mpatha
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3755]: + grep -qw /dev/mapper/myluksdev
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3754]: + lsblk -pno NAME /dev/mapper/mpatha
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + echo ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok /dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: + . /var/opt/kola/extdata/luks-test.sh
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ . /var/opt/kola/extdata/commonlib.sh
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: +++ IFS=' '
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: +++ read -r -a cmdline
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3757]: +++ findmnt -nvr /sysroot -o SOURCE
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ srcdev=/dev/mapper/myluksdev
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ [[ /dev/mapper/myluksdev == /dev/mapper/myluksdev ]]
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3758]: +++ lsblk -o TYPE /dev/mapper/myluksdev --noheadings
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ blktype=crypt
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ [[ crypt == crypt ]]
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3759]: +++ findmnt -nvr /sysroot -o FSTYPE
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ fstype=xfs
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ [[ xfs == xfs ]]
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ ok 'source is XFS on LUKS device'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ echo ok 'source is XFS on LUKS device'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok source is XFS on LUKS device
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3760]: +++ findmnt /sysroot -no OPTIONS
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ rootflags=ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ grep prjquota
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3761]: ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ ok 'root mounted with prjquota'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ echo ok 'root mounted with prjquota'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok root mounted with prjquota
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3762]: +++ dmsetup table myluksdev
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ table='0 32720897 crypt aes-xts-plain64 :64:logon:cryptsetup:11cddb8c-c049-4454-b043-986e604a40e4-d0 0 253:4 32768 2 allow_discards no_read_workqueue'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ grep -q allow_discards
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ grep -q no_read_workqueue
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ ok 'discard and custom option enabled for root LUKS'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ echo ok 'discard and custom option enabled for root LUKS'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok discard and custom option enabled for root LUKS
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ grep -q rd.multipath=default /proc/cmdline
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ expected_what=/dev/disk/by-label/dm-mpath-boot
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3767]: ++ grep -q What=/dev/disk/by-label/dm-mpath-boot
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3766]: ++ systemctl cat boot.mount
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ echo ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ok boot mounted by /dev/disk/by-label/dm-mpath-boot
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ '[' '!' -e /run/ignition-ostree-growfs.stamp ']'
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3751]: ++ /tmp/autopkgtest-reboot rebooted
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3768]: + /usr/local/bin/kolet reboot-request rebooted
Dec 03 10:01:59 qemu0 kola-runext-test.sh[3768]: + reboot
-- Boot b883d68b3bec408780c3901ca29f539e --
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + . /var/opt/kola/extdata/commonlib.sh
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ IFS=' '
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ read -r -a cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1566]: ++ findmnt -nvr /sysroot -o SOURCE
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + srcdev=/dev/mapper/myluksdev
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + parent_device=/dev/mapper/mpatha
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1568]: + grep -qw /dev/mapper/myluksdev
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1567]: + lsblk -pno NAME /dev/mapper/mpatha
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + echo ok '/dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok /dev/mapper/myluksdev device is part of the parent /dev/mapper/mpatha device
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: + . /var/opt/kola/extdata/luks-test.sh
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ . /var/opt/kola/extdata/commonlib.sh
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: +++ IFS=' '
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: +++ read -r -a cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1570]: +++ findmnt -nvr /sysroot -o SOURCE
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ srcdev=/dev/mapper/myluksdev
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ [[ /dev/mapper/myluksdev == /dev/mapper/myluksdev ]]
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1571]: +++ lsblk -o TYPE /dev/mapper/myluksdev --noheadings
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ blktype=crypt
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ [[ crypt == crypt ]]
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1572]: +++ findmnt -nvr /sysroot -o FSTYPE
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ fstype=xfs
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ [[ xfs == xfs ]]
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'source is XFS on LUKS device'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'source is XFS on LUKS device'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok source is XFS on LUKS device
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1573]: +++ findmnt /sysroot -no OPTIONS
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ rootflags=ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep prjquota
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1574]: ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'root mounted with prjquota'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'root mounted with prjquota'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok root mounted with prjquota
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1575]: +++ dmsetup table myluksdev
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ table='0 32720897 crypt aes-xts-plain64 :64:logon:cryptsetup:11cddb8c-c049-4454-b043-986e604a40e4-d0 0 253:4 32768 2 allow_discards no_read_workqueue'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep -q allow_discards
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep -q no_read_workqueue
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'discard and custom option enabled for root LUKS'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'discard and custom option enabled for root LUKS'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok discard and custom option enabled for root LUKS
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep -q rd.multipath=default /proc/cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ expected_what=/dev/disk/by-label/dm-mpath-boot
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1580]: ++ grep -q What=/dev/disk/by-label/dm-mpath-boot
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1579]: ++ systemctl cat boot.mount
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'boot mounted by /dev/disk/by-label/dm-mpath-boot'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok boot mounted by /dev/disk/by-label/dm-mpath-boot
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ case "${AUTOPKGTEST_REBOOT_MARK:-}" in
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep root=UUID= /proc/cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1581]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/rhcos-77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/vmlinuz-5.14.0-427.48.1.el9_4.ppc64le rw ostree=/ostree/boot.1/rhcos/77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=11cddb8c-c049-4454-b043-986e604a40e4=myluksdev root=UUID=de767d2d-4d14-4cb7-a34e-b245c707f150 rw rootflags=prjquota boot=UUID=15f94f74-92c4-40d9-a595-2d3df04138b9
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep rd.luks.name= /proc/cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1582]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/rhcos-77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/vmlinuz-5.14.0-427.48.1.el9_4.ppc64le rw ostree=/ostree/boot.1/rhcos/77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=11cddb8c-c049-4454-b043-986e604a40e4=myluksdev root=UUID=de767d2d-4d14-4cb7-a34e-b245c707f150 rw rootflags=prjquota boot=UUID=15f94f74-92c4-40d9-a595-2d3df04138b9
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'found root kargs'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'found root kargs'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok found root kargs
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ grep boot=UUID= /proc/cmdline
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1583]: BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/rhcos-77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/vmlinuz-5.14.0-427.48.1.el9_4.ppc64le rw ostree=/ostree/boot.1/rhcos/77cb2ccc87d9ebbb587c16780afffc8e73153a955f9373f891bb59624927dd49/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default rd.luks.name=11cddb8c-c049-4454-b043-986e604a40e4=myluksdev root=UUID=de767d2d-4d14-4cb7-a34e-b245c707f150 rw rootflags=prjquota boot=UUID=15f94f74-92c4-40d9-a595-2d3df04138b9
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ ok 'found boot karg'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ++ echo ok 'found boot karg'
Dec 03 10:02:30 qemu0 kola-runext-test.sh[1564]: ok found boot karg
However FCOS, which was previously functional, is now failing to boot as it is throwing the below error.
[coreos-assembler]$ cosa kola run -p qemu --qemu-image ../test/fedora-coreos-41.20241203.dev.0-qemu.ppc64le.qcow2 -E src/config/ 'ext.config.root-reprovision.luks.multipath'
kola run -p qemu --qemu-image ../test/fedora-coreos-41.20241203.dev.0-qemu.ppc64le.qcow2 -E src/config/ ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__ Will warn on failure for kola test pattern "coreos.boot-mirror*":
__ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN ext.config.root-reprovision.luks.multipath
--- FAIL: ext.config.root-reprovision.luks.multipath (72.18s)
harness.go:1823: mach.Start() failed: machine aa6ae511-d5c1-421e-b957-8c3276a31782 entered emergency.target in initramfs
FAIL, output in tmp/kola
Error: harness: test suite failed
2024-12-03T09:49:36Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1
FCOS console logs :
[..snip..]
[ 28.592836] ignition[1596]: disks: createLuks: op(3): [started] creating "myluksdev"
[ 34.831451] ignition[1596]: disks: createLuks: op(3): [finished] creating "myluksdev"
[ 34.831931] ignition[1596]: disks: createLuks: op(4): [started] opening luks device myluksdev
[ 37.546373] ignition[1596]: disks: createLuks: op(4): [finished] opening luks device myluksdev
[ 37.549355] ignition[1596]: disks: createLuks: op(5): [started] Clevis bind
[ 42.681896] ignition[1596]: disks: createLuks: op(5): [finished] Clevis bind
[ 42.682286] ignition[1596]: disks: createLuks: op(6): [started] closing clevis luks device myluksdev
[ 42.750229] ignition[1596]: disks: createLuks: op(6): [finished] closing clevis luks device myluksdev
[ 42.750533] ignition[1596]: disks: createLuks: op(7): [started] reopening clevis luks device myluksdev
[ 45.570754] ignition[1596]: disks: createLuks: op(7): [finished] reopening clevis luks device myluksdev
[ 45.571009] ignition[1596]: disks: createLuks: op(8): [started] removing key file for myluksdev
[ 47.850124] ignition[1596]: disks: createLuks: op(8): [finished] removing key file for myluksdev
[ 47.850353] ignition[1596]: disks: createLuks: op(9): [started] waiting for triggered uevent
[ 47.950205] ignition[1596]: disks: createLuks: op(9): [finished] waiting for triggered uevent
[ 47.950423] ignition[1596]: disks: createFilesystems: op(a): [started] waiting for devices [/dev/mapper/myluksdev]
[ 47.978127] ignition[1596]: disks: createFilesystems: op(a): [finished] waiting for devices [/dev/mapper/myluksdev]
[ 47.978366] ignition[1596]: disks: createFilesystems: created device alias for "/dev/mapper/myluksdev": "/run/ignition/dev_aliases/dev/mapper/myluksdev" -> "/dev/dm-5"
[ 47.978550] ignition[1596]: disks: createFilesystems: op(b): [started] determining filesystem type of "/dev/mapper/myluksdev"
[ 48.084046] ignition[1596]: disks: createFilesystems: op(b): [finished] determining filesystem type of "/dev/mapper/myluksdev"
[ 48.084285] ignition[1596]: disks: createFilesystems: found filesystem at "/dev/mapper/myluksdev" with uuid "" and label ""
[ 48.084450] ignition[1596]: disks: createFilesystems: op(c): [started] wiping filesystem signatures from "/run/ignition/dev_aliases/dev/mapper/myluksdev"
[ 48.129081] ignition[1596]: disks: createFilesystems: op(c): [finished] wiping filesystem signatures from "/run/ignition/dev_aliases/dev/mapper/myluksdev"
[ 48.129320] ignition[1596]: disks: createFilesystems: op(d): [started] creating "xfs" filesystem on "/run/ignition/dev_aliases/dev/mapper/myluksdev"
[ 48.901003] ignition[1596]: disks: createFilesystems: op(d): [finished] creating "xfs" filesystem on "/run/ignition/dev_aliases/dev/mapper/myluksdev"
[ 48.901450] ignition[1596]: disks: createFilesystems: op(e): [started] waiting for triggered uevent
[ 49.014476] ignition[1596]: disks: createFilesystems: op(e): [finished] waiting for triggered uevent
[ 49.014740] ignition[1596]: disks: disks passed
[ 49.014956] ignition[1596]: Ignition finished successfully
[ 49.022491] systemd[1]: Finished ignition-disks.service - Ignition (disks).
[ 49.022944] systemd[1]: Reached target initrd-root-device.target - Initrd Root Device.
[ 49.024331] systemd[1]: Starting coreos-ignition-unique-boot.service - CoreOS Ignition Ensure Unique Boot Filesystem...
[ 49.556492] systemd[1]: Finished coreos-ignition-unique-boot.service - CoreOS Ignition Ensure Unique Boot Filesystem.
[ 49.558446] systemd[1]: ignition-ostree-uuid-root.service - Ignition OSTree: Regenerate Filesystem UUID (root) was skipped because of an unmet condition check (ConditionPathExists=!/run/ignition-ostree-transposefs).
[ 49.562012] systemd[1]: Starting ignition-ostree-growfs.service - Ignition OSTree: Grow Root Filesystem...
[ 49.588122] ignition-ostree-growfs[2335]: Expected /dev/disk/by-label/root to point to /dev/dm-5, but points to /dev/dm-4; triggering udev
[ 49.671835] ignition-ostree-growfs[2370]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
[ 49.672049] ignition-ostree-growfs[2370]: dmesg(1) may have more information after failed mount system call.
[ 49.672639] systemd[1]: ignition-ostree-growfs.service: Main process exited, code=exited, status=32/n/a
[ 49.673019] systemd[1]: ignition-ostree-growfs.service: Failed with result 'exit-code'.
[ 49.673442] systemd[1]: Failed to start ignition-ostree-growfs.service - Ignition OSTree: Grow Root Filesystem.
[ 49.673696] systemd[1]: Dependency failed for ignition-complete.target - Ignition Complete.
[ 49.673868] systemd[1]: Dependency failed for initrd.target - Initrd Default Target.
[ 49.674362] systemd[1]: initrd.target: Job initrd.target/start failed with result 'dependency'.
[ 49.674488] systemd[1]: initrd.target: Triggering OnFailure= dependencies.
[ 49.674965] systemd[1]: ignition-complete.target: Job ignition-complete.target/start failed with result 'dependency'.
[..snip..]
[ 47.210473] ignition-ostree-growfs[2382]: Expected /dev/disk/by-label/root to point to /dev/dm-5, but points to /dev/dm-4; triggering udev
[ 47.267611] ignition-ostree-growfs[2417]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
[ 47.269927] ignition-ostree-growfs[2417]: dmesg(1) may have more information after failed mount system call.
Hmm, definitely odd. This might be some weird udev issue. And this used to pass... so possibly some race that became more racy? Not sure.
I would try to reproduce this manually (do a cosa run -c, passing in the same Ignition config, and --qemu-multipath). Then you can poke around a bit in the emergency shell. E.g. lsblk would be useful.
I have manually reproduced this issue using below commands with the same ignition file.
# cosa run --qemu-image builds/latest/ppc64le/fedora-coreos-41.20241209.dev.0-qemu.ppc64le.qcow2 --qemu-multipath --kargs 'rd.multipath=default' -c -m 4096 -i ../test.ign
[..snip..]
Dec 09 10:22:02 systemd[1]: Failed to mount sysroot.mount - /sysroot.
Dec 09 10:22:01 systemd[1]: Starting ignition-ostree-growfs.service - Ignition OSTree: Grow Root Filesystem...
Dec 09 10:22:01 ignition-ostree-growfs[3826]: Expected /dev/disk/by-label/root to point to /dev/dm-5, but points to /dev/dm-4; triggering udev
Dec 09 10:22:01 ignition-ostree-growfs[3861]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
Dec 09 10:22:01 ignition-ostree-growfs[3861]: dmesg(1) may have more information after failed mount system call.
Dec 09 10:22:01 systemd[1]: ignition-ostree-growfs.service: Main process exited, code=exited, status=32/n/a
Dec 09 10:22:01 systemd[1]: ignition-ostree-growfs.service: Failed with result 'exit-code'.
Dec 09 10:22:01 systemd[1]: Failed to start ignition-ostree-growfs.service - Ignition OSTree: Grow Root Filesystem.
Dec 09 10:22:01 systemd[1]: Starting ignition-ostree-transposefs-restore.service - Ignition OSTree: Restore Partitions...
Dec 09 10:22:01 ignition-ostree-transposefs[3864]: Restoring rootfs from RAM...
Dec 09 10:22:01 ignition-ostree-transposefs[3864]: Expected /dev/disk/by-label/root to point to /dev/dm-5, but points to /dev/dm-4; triggering udev
Dec 09 10:22:01 ignition-ostree-transposefs[3864]: Mounting /dev/disk/by-label/root rw (/dev/dm-4) to /sysroot
Dec 09 10:22:01 ignition-ostree-transposefs[3906]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
Dec 09 10:22:01 ignition-ostree-transposefs[3906]: dmesg(1) may have more information after failed mount system call.
Dec 09 10:22:01 systemd[1]: ignition-ostree-transposefs-restore.service: Main process exited, code=exited, status=32/n/a
Dec 09 10:22:01 systemd[1]: ignition-ostree-transposefs-restore.service: Failed with result 'exit-code'.
Dec 09 10:22:01 systemd[1]: Failed to start ignition-ostree-transposefs-restore.service - Ignition OSTree: Restore Partitions.
Dec 09 10:22:01 systemd[1]: ignition-ostree-transposefs-restore.service: Triggering OnFailure= dependencies.
Generating "/run/initramfs/rdsosreport.txt"
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.
Press Enter for maintenance
(or press Control-D to continue):
:/root# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 10G 0 disk
|-sda1 8:1 0 4M 0 part
|-sda2 8:2 0 1M 0 part
|-sda3 8:3 0 384M 0 part
|-sda4 8:4 0 2.2G 0 part
`-mpatha 253:0 0 10G 0 mpath
|-mpatha1 253:1 0 4M 0 part
|-mpatha2 253:2 0 1M 0 part
|-mpatha3 253:3 0 384M 0 part
`-mpatha4 253:4 0 2.2G 0 part
`-myluksdev 253:5 0 2.2G 0 crypt
sdb 8:16 0 10G 0 disk
|-sdb1 8:17 0 4M 0 part
|-sdb2 8:18 0 1M 0 part
|-sdb3 8:19 0 384M 0 part
|-sdb4 8:20 0 2.2G 0 part
`-mpatha 253:0 0 10G 0 mpath
|-mpatha1 253:1 0 4M 0 part
|-mpatha2 253:2 0 1M 0 part
|-mpatha3 253:3 0 384M 0 part
`-mpatha4 253:4 0 2.2G 0 part
`-myluksdev 253:5 0 2.2G 0 crypt
However, it is booted successfully with the below config.
# cat tmp.bu
variant: fcos
version: 1.4.0
boot_device:
luks:
tpm2: true
---
# cosa run --qemu-image builds/latest/ppc64le/fedora-coreos-41.20241209.dev.0-qemu.ppc64le.qcow2 --qemu-multipath --kargs 'rd.multipath=default' -c -m 4096 -B tmp.bu
[..snip..]
[ 80.761209] systemd[1]: Starting systemd-hwdb-update.service - Rebuild Hardware Database...
[ 80.764083] systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables...
[ 80.769153] systemd[1]: Starting systemd-tmpfiles-setup-dev-early.service - Create Static Device Nodes in /dev gracefully...
[ 80.772613] systemd[1]: Mounted sys-fs-fuse-connections.mount - FUSE Control File System.
Fedora CoreOS 41.20241209.dev.0
Kernel 6.11.10-300.fc41.ppc64le on an ppc64le (hvc0)
SSH host key: SHA256:uJ/jYOQwgeH8/i/fjqY69mLbhr17zz28uRb7C9nwvKE (ECDSA)
SSH host key: SHA256:6jp/Kz7HX6HI1dpjQ2FeleL3iIvDr5QFhAS2u1SahLA (ED25519)
SSH host key: SHA256:KDG4F8xLPoR+nzBs6tEccy27qN1TYMSjsCEXe9P49lo (RSA)
enp0s3: 10.0.2.15 fe80::bc4:f482:8844:c646
Ignition: ran on 2024/12/09 12:36:52 UTC (this boot)
Ignition: user-provided config was applied
No SSH authorized keys provided by Ignition or Afterburn
cosa-devsh login: core (automatic login)
Fedora CoreOS 41.20241209.dev.0
[core@cosa-devsh ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 4.3M 1 loop
sda 8:0 0 10G 0 disk
└─mpatha 253:0 0 10G 0 mpath
├─mpatha1 253:1 0 4M 0 part
├─mpatha2 253:2 0 1M 0 part
├─mpatha3 253:3 0 384M 0 part /boot
└─mpatha4 253:4 0 9.6G 0 part
└─root 253:5 0 9.6G 0 crypt /var
/sysroot/ostree/deploy/fedora-coreos/var
/etc
/sysroot
sdb 8:16 0 10G 0 disk
└─mpatha 253:0 0 10G 0 mpath
├─mpatha1 253:1 0 4M 0 part
├─mpatha2 253:2 0 1M 0 part
├─mpatha3 253:3 0 384M 0 part /boot
└─mpatha4 253:4 0 9.6G 0 part
└─root 253:5 0 9.6G 0 crypt /var
/sysroot/ostree/deploy/fedora-coreos/var
/etc
/sysroot
@dustymabe I have run the coreos-builds-bisect.py with stable and testing-devel builds and found the failure introduced build.
With stable builds
[coreos-assembler]$ cat test.sh
#!/bin/bash
set -eux -o pipefail
build=$1
cosa buildfetch --force --stream=stable --build=$build --artifact=qemu
cosa decompress --build=$build
cosa kola run --build=$build 'ext.config.root-reprovision.luks.multipath'
[coreos-assembler]$ chmod 777 test.sh
[coreos-assembler]$ cosa buildfetch --force --stream=stable
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/41.20241109.3.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/41.20241109.3.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/41.20241109.3.0/ppc64le/ostree-commit-object
[coreos-assembler]$ ./coreos-builds-bisect.py --bad 41.20241109.3.0 --good 40.20240920.3.0 --testscript=./test.sh
Using data file at: ./coreos-builds-bisect-data.json
Targetting architecture: ppc64le
Executing test for new build: 40.20241019.3.0. ~2 steps left
+ build=40.20241019.3.0
+ cosa buildfetch --force --stream=stable --build=40.20241019.3.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241019.3.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241019.3.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241019.3.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241019.3.0/ppc64le/fedora-coreos-40.20241019.3.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=40.20241019.3.0
Targeting build: 40.20241019.3.0
Skipping missing arch: x86_64
Skipping missing arch: s390x
Skipping missing arch: aarch64
Uncompressing: builds/40.20241019.3.0/ppc64le
2024-12-11 12:53:32,129 INFO - Running command: ['xz', '-dc', '-T16', 'builds/40.20241019.3.0/ppc64le/fedora-coreos-40.20241019.3.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-40.20241019.3.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/40.20241019.3.0/ppc64le/meta.json
+ cosa kola run --build=40.20241019.3.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 40.20241019.3.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__ Will warn on failure for kola test pattern "coreos.boot-mirror*":
__ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN ext.config.root-reprovision.luks.multipath
--- FAIL: ext.config.root-reprovision.luks.multipath (160.98s)
harness.go:1823: mach.Start() failed: machine db54928d-6b77-4ee6-bd36-8e5b494c7f77 entered emergency.target in initramfs
FAIL, output in tmp/kola
Error: harness: test suite failed
2024-12-11T12:56:24Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1
40.20241019.3.0 failed
Executing test for new build: 40.20241006.3.0. ~1 steps left
+ build=40.20241006.3.0
+ cosa buildfetch --force --stream=stable --build=40.20241006.3.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241006.3.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241006.3.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241006.3.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/40.20241006.3.0/ppc64le/fedora-coreos-40.20241006.3.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=40.20241006.3.0
Targeting build: 40.20241006.3.0
Skipping missing arch: x86_64
Skipping missing arch: aarch64
Uncompressing: builds/40.20241006.3.0/ppc64le
2024-12-11 12:56:55,356 INFO - Running command: ['xz', '-dc', '-T16', 'builds/40.20241006.3.0/ppc64le/fedora-coreos-40.20241006.3.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-40.20241006.3.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/40.20241006.3.0/ppc64le/meta.json
Skipping missing arch: s390x
+ cosa kola run --build=40.20241006.3.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 40.20241006.3.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__ Will warn on failure for kola test pattern "coreos.boot-mirror*":
__ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN ext.config.root-reprovision.luks.multipath
--- PASS: ext.config.root-reprovision.luks.multipath (277.93s)
PASS, output in tmp/kola
40.20241006.3.0 passed
BISECT TEST RESULTS:
Last known good build: 40.20241006.3.0
First known bad build: 40.20241019.3.0
[coreos-assembler]$
With testing-devel builds:
[coreos-assembler]$ ./coreos-builds-bisect.py --bad 41.20241108.20.0 --good 40.20241013.20.0 --testscript=./test.sh
[..snip..]
Executing test for new build: 41.20241030.20.0. ~4 steps left
+ build=41.20241030.20.0
+ cosa buildfetch --force --stream=testing-devel --build=41.20241030.20.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/41.20241030.20.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/41.20241030.20.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/41.20241030.20.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/41.20241030.20.0/ppc64le/fedora-coreos-41.20241030.20.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=41.20241030.20.0
Targeting build: 41.20241030.20.0
Skipping missing arch: x86_64
Skipping missing arch: s390x
Skipping missing arch: aarch64
Uncompressing: builds/41.20241030.20.0/ppc64le
2024-12-11 14:16:51,398 INFO - Running command: ['xz', '-dc', '-T16', 'builds/41.20241030.20.0/ppc64le/fedora-coreos-41.20241030.20.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-41.20241030.20.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/41.20241030.20.0/ppc64le/meta.json
+ cosa kola run --build=41.20241030.20.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 41.20241030.20.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__ Will warn on failure for kola test pattern "coreos.boot-mirror*":
__ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN ext.config.root-reprovision.luks.multipath
--- FAIL: ext.config.root-reprovision.luks.multipath (121.29s)
harness.go:1823: mach.Start() failed: machine 22e6b73b-ae84-411f-ac32-7150350eb73a entered emergency.target in initramfs
FAIL, output in tmp/kola
Error: harness: test suite failed
2024-12-11T14:19:02Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1
41.20241030.20.0 failed
Executing test for new build: 40.20241019.20.0. ~3 steps left
+ build=40.20241019.20.0
+ cosa buildfetch --force --stream=testing-devel --build=40.20241019.20.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241019.20.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241019.20.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241019.20.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241019.20.0/ppc64le/fedora-coreos-40.20241019.20.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=40.20241019.20.0
Targeting build: 40.20241019.20.0
Skipping missing arch: x86_64
Skipping missing arch: s390x
Skipping missing arch: aarch64
Uncompressing: builds/40.20241019.20.0/ppc64le
2024-12-11 14:19:39,005 INFO - Running command: ['xz', '-dc', '-T16', 'builds/40.20241019.20.0/ppc64le/fedora-coreos-40.20241019.20.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-40.20241019.20.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/40.20241019.20.0/ppc64le/meta.json
+ cosa kola run --build=40.20241019.20.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 40.20241019.20.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__ Will warn on failure for kola test pattern "coreos.boot-mirror*":
__ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN ext.config.root-reprovision.luks.multipath
--- FAIL: ext.config.root-reprovision.luks.multipath (131.85s)
harness.go:1823: mach.Start() failed: machine a3a6a267-bc52-43ea-931d-cc161287c30f entered emergency.target in initramfs
FAIL, output in tmp/kola
Error: harness: test suite failed
2024-12-11T14:22:02Z cli: harness: test suite failed
failed to execute cmd-kola: exit status 1
40.20241019.20.0 failed
Executing test for new build: 40.20241017.20.0. ~1 steps left
+ build=40.20241017.20.0
+ cosa buildfetch --force --stream=testing-devel --build=40.20241017.20.0 --artifact=qemu
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/builds.json
Updated builds/builds.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241017.20.0/ppc64le/meta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241017.20.0/ppc64le/commitmeta.json
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241017.20.0/ppc64le/ostree-commit-object
Fetching: https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/40.20241017.20.0/ppc64le/fedora-coreos-40.20241017.20.0-qemu.ppc64le.qcow2.xz
+ cosa decompress --build=40.20241017.20.0
Targeting build: 40.20241017.20.0
Skipping missing arch: x86_64
Skipping missing arch: aarch64
Uncompressing: builds/40.20241017.20.0/ppc64le
2024-12-11 14:22:39,418 INFO - Running command: ['xz', '-dc', '-T16', 'builds/40.20241017.20.0/ppc64le/fedora-coreos-40.20241017.20.0-qemu.ppc64le.qcow2.xz']
Uncompressed: fedora-coreos-40.20241017.20.0-qemu.ppc64le.qcow2
Skipped uncompressing artifacts: ostree powervs live-iso live-kernel live-initramfs live-rootfs
Skipped missing artifacts: oci-manifest openstack metal metal4k
Updated: builds/40.20241017.20.0/ppc64le/meta.json
Skipping missing arch: s390x
+ cosa kola run --build=40.20241017.20.0 ext.config.root-reprovision.luks.multipath
kola -p qemu --build 40.20241017.20.0 run ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__ Will warn on failure for kola test pattern "coreos.boot-mirror*":
__ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN ext.config.root-reprovision.luks.multipath
--- PASS: ext.config.root-reprovision.luks.multipath (272.80s)
PASS, output in tmp/kola
40.20241017.20.0 passed
BISECT TEST RESULTS:
Last known good build: 40.20241017.20.0
First known bad build: 40.20241019.20.0
BISECT TEST RESULTS: Last known good build: 40.20241017.20.0 First known bad build: 40.20241019.20.0
That represents this diff:
[coreos-assembler]$ cosa diff --from=40.20241017.20.0 --to=40.20241019.20.0 --rpms
Extracting c1fa8de1dfea7a0ab99f5f86f198edfa5967ec9ab98f1a826abd3692de096a81
Downloading... 1
3134 metadata, 20030 content objects imported; 1.7 GB content written
Extracting b647966fbbcc704aea7f9b735c462911a3837078c64a4c7314d41dbd3543b087
Downloading... 7
388 metadata, 2295 content objects imported; 593.7 MB content written
2024-12-11 19:35:53,684 INFO - Running command: ['rpm-ostree', 'db', 'diff', '--repo', 'tmp/repo', 'c1fa8de1dfea7a0ab99f5f86f198edfa5967ec9ab98f1a826abd3692de096a81', 'b647966fbbcc704aea7f9b735c462911a3837078c64a4c7314d41dbd3543b087']
ostree diff commit from: c1fa8de1dfea7a0ab99f5f86f198edfa5967ec9ab98f1a826abd3692de096a81
ostree diff commit to: b647966fbbcc704aea7f9b735c462911a3837078c64a4c7314d41dbd3543b087
Upgraded:
amd-gpu-firmware 20240909-1.fc40 -> 20241017-1.fc40
atheros-firmware 20240909-1.fc40 -> 20241017-1.fc40
brcmfmac-firmware 20240909-1.fc40 -> 20241017-1.fc40
bsdtar 3.7.2-4.fc40 -> 3.7.2-7.fc40
containers-common 5:0.60.4-1.fc40 -> 5:0.60.4-2.fc40
containers-common-extra 5:0.60.4-1.fc40 -> 5:0.60.4-2.fc40
device-mapper 1.02.197-1.fc40 -> 1.02.199-1.fc40
device-mapper-event 1.02.197-1.fc40 -> 1.02.199-1.fc40
device-mapper-event-libs 1.02.197-1.fc40 -> 1.02.199-1.fc40
device-mapper-libs 1.02.197-1.fc40 -> 1.02.199-1.fc40
fwupd 1.9.25-1.fc40 -> 1.9.26-1.fc40
intel-gpu-firmware 20240909-1.fc40 -> 20241017-1.fc40
libarchive 3.7.2-4.fc40 -> 3.7.2-7.fc40
libjcat 0.2.1-2.fc40 -> 0.2.2-1.fc40
libuv 1:1.49.0-1.fc40 -> 1:1.49.1-1.fc40
libxmlb 0.3.19-2.fc40 -> 0.3.21-1.fc40
linux-firmware 20240909-1.fc40 -> 20241017-1.fc40
linux-firmware-whence 20240909-1.fc40 -> 20241017-1.fc40
lvm2 2.03.23-1.fc40 -> 2.03.25-1.fc40
lvm2-libs 2.03.23-1.fc40 -> 2.03.25-1.fc40
mt7xxx-firmware 20240909-1.fc40 -> 20241017-1.fc40
nvidia-gpu-firmware 20240909-1.fc40 -> 20241017-1.fc40
nxpwireless-firmware 20240909-1.fc40 -> 20241017-1.fc40
qed-firmware 20240909-1.fc40 -> 20241017-1.fc40
realtek-firmware 20240909-1.fc40 -> 20241017-1.fc40
shadow-utils 2:4.15.1-3.fc40 -> 2:4.15.1-4.fc40
shadow-utils-subid 2:4.15.1-3.fc40 -> 2:4.15.1-4.fc40
tiwilink-firmware 20240909-1.fc40 -> 20241017-1.fc40
I recently attempted to use the latest test-devel code, I just only reverted to the below RPMs that were present in the good build(40.20241017.20.0) and built the qcow2 image.
Change commit link
device-mapper 1.02.197-1.fc40
device-mapper-event 1.02.197-1.fc40
device-mapper-event-libs 1.02.197-1.fc40
device-mapper-libs 1.02.197-1.fc40
lvm2 2.03.23-1.fc40
lvm2-libs 2.03.23-1.fc40
And the test is passed with the newly created image.
[coreos-assembler]$ cosa kola run -p qemu --qemu-image /home/jenkins/agent/workspace/debug-pod/test-build/builds/41.20250212.dev.0/ppc64le/fedora-coreos-41.20250212.dev.0-qemu.ppc64le.qcow2 'ext.config.root-reprovision.luks.multipath'
kola run -p qemu --qemu-image /home/jenkins/agent/workspace/debug-pod/test-build/builds/41.20250212.dev.0/ppc64le/fedora-coreos-41.20250212.dev.0-qemu.ppc64le.qcow2 ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__ Will warn on failure for kola test pattern "coreos.boot-mirror*":
__ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN ext.config.root-reprovision.luks.multipath
--- PASS: ext.config.root-reprovision.luks.multipath (356.36s)
@Poorna-Gottimukkula1 Can you retest this on top of https://github.com/coreos/fedora-coreos-config/pull/3375 ?
Yes, I retested with above patch with latest build, and the Fedora VM booted successfully.
# cosa run --qemu-image fedora-coreos-41.20250223.20.1-qemu.ppc64le.qcow2 --qemu-multipath --kargs 'rd.multipath=default' -c -m 4096 -i ../fedora-coreos-config/tests/kola/root-reprovision/luks/config.ign
[..snip..]
[ 218.733207] systemd[1]: Starting systemd-journald.service - Journal Service...
[ 218.744358] systemd[1]: Starting systemd-modules-load.service - Load Kernel Modules...
[ 218.747095] systemd[1]: Starting systemd-network-generator.service - Generate network units from Kernel command line...
[ 218.750522] systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
[ 218.763127] systemd[1]: Starting systemd-udev-load-credentials.service - Load udev Rules from Credentials...
[ 218.770958] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...
[ 218.785739] systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
[ 218.790710] systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.
[ 218.793066] systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
[ 218.795633] systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
[ 218.796967] systemd[1]: Mounted tmp.mount - Temporary Directory /tmp.
[ 218.803843] systemd[1]: Finished coreos-printk-quiet.service - CoreOS: Set printk To Level 4 (warn).
[ 218.804904] systemd[1]: Finished kmod-static-nodes.service - Create List of Static Device Nodes.
[ 218.805759] systemd[1]: [email protected]: Deactivated successfully.
[ 218.806340] systemd[1]: Finished [email protected] - Load Kernel Module configfs.
[ 218.807201] systemd[1]: modprobe@dm_multipath.service: Deactivated successfully.
[ 218.807791] systemd[1]: Finished modprobe@dm_multipath.service - Load Kernel Module dm_multipath.
[ 218.808662] systemd[1]: [email protected]: Deactivated successfully.
[ 218.809225] systemd[1]: Finished [email protected] - Load Kernel Module drm.
[ 218.810057] systemd[1]: modprobe@efi_pstore.service: Deactivated successfully.
[ 218.810678] systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
[ 218.811537] systemd[1]: [email protected]: Deactivated successfully.
[ 218.812097] systemd[1]: Finished [email protected] - Load Kernel Module fuse.
[ 218.812967] systemd[1]: Finished systemd-remount-fs.service - Remount Root and Kernel File Systems.
[ 218.846415] systemd[1]: Mounting sys-fs-fuse-connections.mount - FUSE Control File System...
[ 218.875083] systemd[1]: Starting multipathd.service - Device-Mapper Multipath Device Controller...
[ 218.875446] systemd[1]: iscsi-onboot.service - Special handling of early boot iSCSI sessions was skipped because of an unmet condition check (ConditionDirectoryNotEmpty=/sys/class/iscsi_session).
[ 218.893471] systemd[1]: Starting systemd-hwdb-update.service - Rebuild Hardware Database...
[ 218.913178] systemd[1]: Starting systemd-tmpfiles-setup-dev-early.service - Create Static Device Nodes in /dev gracefully...
[ 218.915452] systemd[1]: Finished systemd-modules-load.service - Load Kernel Modules.
[ 218.920904] systemd-journald[4658]: Collecting audit messages is enabled.
[ 218.923904] systemd[1]: Finished systemd-network-generator.service - Generate network units from Kernel command line.
[ 218.926122] systemd[1]: Finished systemd-udev-load-credentials.service - Load udev Rules from Credentials.
[ 218.928034] systemd[1]: Mounted sys-fs-fuse-connections.mount - FUSE Control File System.
[ 218.928485] systemd[1]: Reached target network-pre.target - Preparation for Network.
[ 218.934572] kauditd_printk_skb: 4 callbacks suppressed
[ 218.934578] audit: type=1130 audit(1740474586.580:16): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-modules-load comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 218.934994] audit: type=1130 audit(1740474586.580:17): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-network-generator comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 218.935301] audit: type=1130 audit(1740474586.580:18): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-udev-load-credentials comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 218.943158] systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables...
[ 218.943534] systemd[1]: Started systemd-journald.service - Journal Service.
[ 218.944379] audit: type=1130 audit(1740474586.600:19): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 218.989340] audit: type=1130 audit(1740474586.640:20): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-cryptsetup@myluksdev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 219.076114] audit: type=1130 audit(1740474586.730:21): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-udev-trigger comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Fedora CoreOS 41.20250223.20.1
Kernel 6.12.15-200.fc41.ppc64le on an ppc64le (hvc0)
SSH host key: SHA256:7PQzD3Ov7Ol5zUznDdVDxUbQdp+/7WPHVFueL59Pt+Q (ECDSA)
SSH host key: SHA256:2nq4Ua+0GZospLT0xO3VwcpgbqNOwl51ROHfp/gW/9o (ED25519)
SSH host key: SHA256:inBzFnzZreZdLZbEH4OxyBBxIitiSHGEaCSBCZG/EZg (RSA)
enp0s3: 10.0.2.15 fec0::92d:95fb:3fa5:eda6
Ignition: ran on 2025/02/25 09:09:34 UTC (this boot)
Ignition: user-provided config was applied
No SSH authorized keys provided by Ignition or Afterburn
cosa-devsh login: core (automatic login)
Last login: Tue Feb 25 09:10:00 on tty1
Fedora CoreOS 41.20250223.20.1
[core@cosa-devsh ~]$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
sda mpath_member
└─mpatha
├─mpatha1
├─mpatha2
├─mpatha3 ext4 1.0 boot f9e417ca-1641-4f29-a08a-5264c4a40efe 179.8M 42% /boot
└─mpatha4 crypto_LUKS 2 luks-root 23116f28-ccd0-4652-b1e2-cea7bec1c56f
└─myluksdev xfs root 46d7a250-2cc1-41dc-8314-cc6c510507fd 7.8G 18% /var
/sysroot/ostree/deploy/fedora-coreos/var
/etc
/sysroot
sdb mpath_member
└─mpatha
├─mpatha1
├─mpatha2
├─mpatha3 ext4 1.0 boot f9e417ca-1641-4f29-a08a-5264c4a40efe 179.8M 42% /boot
└─mpatha4 crypto_LUKS 2 luks-root 23116f28-ccd0-4652-b1e2-cea7bec1c56f
└─myluksdev xfs root 46d7a250-2cc1-41dc-8314-cc6c510507fd 7.8G 18% /var
/sysroot/ostree/deploy/fedora-coreos/var
/etc
/sysroot
[core@cosa-devsh ~]$ cat /proc/cmdline
BOOT_IMAGE=(ieee1275/disk,gpt3)/boot/ostree/fedora-coreos-5c01224aaa04c2c2e68732fa9cfd653e93c79f8d7dab33ad68ed393fa6fc7fc9/vmlinuz-6.12.15-200.fc41.ppc64le rw ignition.firstboot mitigations=auto,nosmt ostree=/ostree/boot.1/fedora-coreos/5c01224aaa04c2c2e68732fa9cfd653e93c79f8d7dab33ad68ed393fa6fc7fc9/0 ignition.platform.id=qemu console=hvc0 console=tty0 rd.multipath=default
FCOS: And the kola test is also passed with latest build.
[coreos-assembler]$ cosa kola run -p qemu --qemu-image fedora-coreos-41.20250223.20.1-qemu.ppc64le.qcow2 -E src/config/ 'ext.config.root-reprovision.luks.multipath'
kola run -p qemu --qemu-image fedora-coreos-41.20250223.20.1-qemu.ppc64le.qcow2 -E src/config/ ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__ Will warn on failure for kola test pattern "coreos.boot-mirror*":
__ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN ext.config.root-reprovision.luks.multipath
--- PASS: ext.config.root-reprovision.luks.multipath (409.08s)
PASS, output in tmp/kola
RHCOS: is also passed.
[coreos-assembler]$ cosa kola run -p qemu --qemu-image rhcos-419.94.202501160007-0-qemu.ppc64le.qcow2 -E src/config/ 'ext.config.root-reprovision.luks.multipath'
kola run -p qemu --qemu-image rhcos-419.94.202501160007-0-qemu.ppc64le.qcow2 -E src/config/ ext.config.root-reprovision.luks.multipath --output-dir tmp/kola
__ Will warn on failure for kola test pattern "coreos.boot-mirror*":
__ https://github.com/coreos/fedora-coreos-tracker/issues/1659
=== RUN ext.config.root-reprovision.luks.multipath
--- PASS: ext.config.root-reprovision.luks.multipath (173.58s)
PASS, output in tmp/kola
cc: @dustymabe @jlebon