coreos-assembler
coreos-assembler copied to clipboard
coreos
kola port clashing for qemu user mode networking
When we run kola in the pipeline we run many many tests in a single run, but only 4 at a time (--parallel=4).
It appears, though, that some qemu processes are hanging around longer. For example:
core@coreos-ppc64le-builder:~$ sudo ss -lutnp | grep qemu
udp UNCONN 0 0 0.0.0.0:32771 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=143))
udp UNCONN 0 0 0.0.0.0:33353 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=163))
udp UNCONN 0 0 0.0.0.0:33513 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=185))
udp UNCONN 0 0 0.0.0.0:35914 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=122))
udp UNCONN 0 0 0.0.0.0:36387 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=152))
udp UNCONN 0 0 0.0.0.0:36766 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=133))
udp UNCONN 0 0 0.0.0.0:36842 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=169))
udp UNCONN 0 0 0.0.0.0:36847 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=108))
udp UNCONN 0 0 0.0.0.0:37014 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=118))
udp UNCONN 0 0 0.0.0.0:37312 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=114))
udp UNCONN 0 0 0.0.0.0:37328 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=156))
udp UNCONN 0 0 0.0.0.0:37600 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=165))
udp UNCONN 0 0 0.0.0.0:37746 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=145))
udp UNCONN 0 0 0.0.0.0:37902 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=107))
udp UNCONN 0 0 0.0.0.0:38864 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=168))
udp UNCONN 0 0 0.0.0.0:38962 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=110))
udp UNCONN 0 0 0.0.0.0:40629 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=112))
udp UNCONN 0 0 0.0.0.0:41679 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=159))
udp UNCONN 0 0 0.0.0.0:41731 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=142))
udp UNCONN 0 0 0.0.0.0:42216 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=170))
udp UNCONN 0 0 0.0.0.0:43546 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=149))
udp UNCONN 0 0 0.0.0.0:44382 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=121))
udp UNCONN 0 0 0.0.0.0:44477 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=146))
udp UNCONN 0 0 0.0.0.0:44969 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=106))
udp UNCONN 0 0 0.0.0.0:45113 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=135))
udp UNCONN 0 0 0.0.0.0:45711 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=30))
udp UNCONN 0 0 0.0.0.0:45852 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=130))
udp UNCONN 0 0 0.0.0.0:46064 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=126))
udp UNCONN 0 0 0.0.0.0:47320 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=154))
udp UNCONN 0 0 0.0.0.0:47657 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=131))
udp UNCONN 0 0 0.0.0.0:48291 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=187))
udp UNCONN 0 0 0.0.0.0:48428 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=148))
udp UNCONN 0 0 0.0.0.0:50286 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=155))
udp UNCONN 0 0 0.0.0.0:50475 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=140))
udp UNCONN 0 0 0.0.0.0:50478 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=120))
udp UNCONN 0 0 0.0.0.0:51084 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=164))
udp UNCONN 0 0 0.0.0.0:51150 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=158))
udp UNCONN 0 0 0.0.0.0:51467 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=160))
udp UNCONN 0 0 0.0.0.0:51639 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=115))
udp UNCONN 0 0 0.0.0.0:51643 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=162))
udp UNCONN 0 0 0.0.0.0:51867 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=138))
udp UNCONN 0 0 0.0.0.0:51911 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=134))
udp UNCONN 0 0 0.0.0.0:52978 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=144))
udp UNCONN 0 0 0.0.0.0:53949 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=171))
udp UNCONN 0 0 0.0.0.0:54155 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=129))
udp UNCONN 0 0 0.0.0.0:54285 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=166))
udp UNCONN 0 0 0.0.0.0:54296 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=167))
udp UNCONN 0 0 0.0.0.0:54434 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=137))
udp UNCONN 0 0 0.0.0.0:55749 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=189))
udp UNCONN 0 0 0.0.0.0:55831 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=113))
udp UNCONN 0 0 0.0.0.0:55904 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=139))
udp UNCONN 0 0 0.0.0.0:56132 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=111))
udp UNCONN 0 0 0.0.0.0:56266 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=182))
udp UNCONN 0 0 0.0.0.0:56434 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=151))
udp UNCONN 0 0 0.0.0.0:56636 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=141))
udp UNCONN 0 0 0.0.0.0:56930 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=153))
udp UNCONN 0 0 0.0.0.0:57442 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=117))
udp UNCONN 0 0 0.0.0.0:57848 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=136))
udp UNCONN 0 0 0.0.0.0:57890 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=150))
udp UNCONN 0 0 0.0.0.0:58370 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=116))
udp UNCONN 0 0 0.0.0.0:58738 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=157))
udp UNCONN 0 0 0.0.0.0:59310 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=161))
udp UNCONN 0 0 0.0.0.0:59466 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=128))
udp UNCONN 0 0 0.0.0.0:60703 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=119))
udp UNCONN 0 0 *:37513 *:* users:(("qemu-system-ppc",pid=1975486,fd=127))
udp UNCONN 0 0 *:39929 *:* users:(("qemu-system-ppc",pid=1975486,fd=125))
udp UNCONN 0 0 *:46748 *:* users:(("qemu-system-ppc",pid=1975486,fd=124))
udp UNCONN 0 0 *:49099 *:* users:(("qemu-system-ppc",pid=1975486,fd=132))
udp UNCONN 0 0 *:50749 *:* users:(("qemu-system-ppc",pid=1975486,fd=123))
udp UNCONN 0 0 *:54827 *:* users:(("qemu-system-ppc",pid=1975486,fd=147))
tcp LISTEN 0 1 127.0.0.1:33359 0.0.0.0:* users:(("qemu-system-ppc",pid=1975455,fd=19))
tcp LISTEN 0 1 127.0.0.1:33229 0.0.0.0:* users:(("qemu-system-ppc",pid=1975987,fd=19))
tcp LISTEN 0 1 127.0.0.1:38673 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=19))
tcp LISTEN 0 1 127.0.0.1:41731 0.0.0.0:* users:(("qemu-system-ppc",pid=1975753,fd=19))
The majority of those are owned by the same process so there might be some opportunity for us to figure out how to clean those up, or maybe they are legitimate somehow?
The bigger problem however is that we had a port clash with the kdump.crash.nfs test. The test requires the use of the "Qemu Host" IP:Port 10.0.2.2:2049 so it can end up clashing if something else happens to be using that port already.
Something like this happened recently in the RHCOS pipeline:
[2025-05-30T14:53:17.900Z] + set -xeuo pipefail
[2025-05-30T14:53:17.900Z] ++ umask
[2025-05-30T14:53:17.900Z] + '[' 0022 = 0000 ']'
[2025-05-30T14:53:17.900Z] + cd /home/jenkins/agent/workspace/build-arch
[2025-05-30T14:53:17.900Z] + cosa kola run --rerun --allow-rerun-success=tags=needs-internet --build=latest --output-dir=/home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola --on-warn-failure-exit-77 --arch=ppc64le '--tag=!reprovision' '--tag=!openshift' --parallel=4
[2025-05-30T14:53:19.782Z] warning: /sys/fs/selinux appears to be mounted but should not be; enabling workaround
[2025-05-30T14:53:19.782Z] kola -p qemu --build latest run --rerun --allow-rerun-success=tags=needs-internet --on-warn-failure-exit-77 --arch=ppc64le --tag=!reprovision --tag=!openshift --parallel=4 --output-dir /home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola
[2025-05-30T14:53:20.708Z] ⏭️ Skipping kola test pattern "ext.config.version.rhaos-pkgs-match-openshift":
[2025-05-30T14:53:20.708Z] 👉 https://issues.redhat.com/browse/OCPBUGS-42688
[2025-05-30T14:53:20.708Z] ⏭️ Skipping kola test pattern "ext.config.shared.multipath.resilient":
[2025-05-30T14:53:20.708Z] 👉 https://github.com/coreos/fedora-coreos-tracker/issues/1937
[2025-05-30T14:53:21.275Z] === RUN ostree.hotfix
[2025-05-30T14:53:21.275Z] === RUN coreos.unique.boot.failure
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.disks.lvmdevices
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.clhm.ignition-warnings
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.containers.quadlet
[2025-05-30T14:53:21.275Z] === RUN coreos.misc.disk.varlibcontainers
[2025-05-30T14:53:21.275Z] === RUN coreos.ignition.mount.disks
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.nameserver
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.boot.grub2-install
[2025-05-30T14:53:21.275Z] === RUN fips.enable
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ntp.chrony.dhcp-propagation
[2025-05-30T14:53:21.275Z] === RUN multipath.day1
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ignition.systemd-enable-units
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ignition.kargs
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.var-mount.simple
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.reboot
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ignition.delete-config
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.mtu-on-bond-ignition
[2025-05-30T14:53:21.275Z] === RUN coreos.ignition.sethostname
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.ifname-karg.everyboot-systemd-link-file
[2025-05-30T14:53:21.275Z] === RUN fips.enable.partitions
[2025-05-30T14:53:21.275Z] === RUN rhcos.sssd
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.no-persist-ip
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.gshadow
[2025-05-30T14:53:21.275Z] === RUN rhcos.selinux.manage
[2025-05-30T14:53:21.275Z] === RUN coreos.unique.boot.ignition.failure
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.podman.rootless-systemd
[2025-05-30T14:53:21.275Z] === RUN non-exclusive-test-bucket-0
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.var-mount.luks
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.force-persist-ip
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ssh.custom-host-key-permissions
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.prefer-ignition-networking
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.groups
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.systemd.network-online
[2025-05-30T14:53:21.276Z] === RUN coreos.selinux.boolean
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.rd-net-timeout-carrier
[2025-05-30T14:53:21.276Z] === RUN ext.config.disk.nvme-compat-symlink
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.boot.bootupd-validate
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.systemd.enable-service
[2025-05-30T14:53:21.276Z] === RUN rpmostree.upgrade-rollback
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.ignition.stable-boot
[2025-05-30T14:53:21.276Z] === RUN kdump.crash.ssh
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.hostname.fallback-hostname
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.failure
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.v2.users
[2025-05-30T14:53:21.276Z] === RUN coreos.auth.verify
[2025-05-30T14:53:21.276Z] === RUN multipath.day2
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.nmstate.policy
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.ignition.remote
[2025-05-30T14:53:21.276Z] === RUN basic
[2025-05-30T14:53:21.276Z] === RUN rpmostree.status
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.team-dhcp-via-ignition
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.mtu-on-bond-kargs
[2025-05-30T14:53:21.276Z] === RUN ext.config.rpm-ostree.replace-rt-kernel
[2025-05-30T14:53:21.276Z] === RUN ostree.sync
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.selinux.enforcing
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.ifname-karg.udev-rule-firstboot-propagation
[2025-05-30T14:53:21.276Z] === RUN ostree.remote
[2025-05-30T14:53:21.276Z] === RUN ext.config.systemd.journal-compat
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.once
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.symlink
[2025-05-30T14:53:21.276Z] === RUN ostree.unlock
[2025-05-30T14:53:21.276Z] === RUN multipath.partition
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.kargs-rd-net
[2025-05-30T14:53:21.276Z] === RUN rhcos.services-disabled
[2025-05-30T14:53:21.276Z] === RUN kdump.crash.nfs
[2025-05-30T14:53:21.276Z] === RUN ext.config.rpm-ostree.startlimit
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.ignition.resource.remote
[2025-05-30T14:53:21.276Z] === RUN rpmostree.install-uninstall
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.kdump.crash
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.no-default-initramfs-net-propagation.bootif
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.systemd.condition-needs-update
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.nmstate.state
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.var-mount.scsi-id
[2025-05-30T14:53:21.276Z] === RUN rhcos.network.multiple-nics
[2025-05-30T14:53:21.276Z] === RUN podman.base
[2025-05-30T14:53:21.276Z] === RUN rhcos.selinux.boolean.persist
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.bridge-static-via-kargs
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.mount.partitions
[2025-05-30T14:53:59.909Z] === RUN ostree.hotfix/unlock
[2025-05-30T14:54:01.789Z] --- PASS: rpmostree.status (40.41s)
[2025-05-30T14:54:03.679Z] --- PASS: ext.config.rpm-ostree.startlimit (42.28s)
[2025-05-30T14:54:03.934Z] --- PASS: ext.config.shared.systemd.network-online (42.65s)
[2025-05-30T14:54:18.756Z] === RUN ostree.hotfix/install
[2025-05-30T14:54:18.756Z] === RUN ostree.hotfix/uninstall
[2025-05-30T14:54:19.010Z] === RUN ostree.hotfix/persist
[2025-05-30T14:54:40.871Z] === RUN basic/MachineID
[2025-05-30T14:54:41.125Z] === RUN basic/RHCOSGrowpart
[2025-05-30T14:54:41.380Z] === RUN basic/PortSSH
[2025-05-30T14:54:41.380Z] === RUN basic/DbusPerms
[2025-05-30T14:54:41.947Z] === RUN basic/ServicesActive
[2025-05-30T14:54:41.947Z] === RUN basic/ReadOnly
[2025-05-30T14:54:42.201Z] === RUN basic/Useradd
[2025-05-30T14:54:44.084Z] --- PASS: ext.config.shared.networking.hostname.fallback-hostname (42.11s)
[2025-05-30T14:54:45.450Z] --- PASS: basic (41.98s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/MachineID (0.14s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/RHCOSGrowpart (0.17s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/PortSSH (0.14s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/DbusPerms (0.47s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/ServicesActive (0.15s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/ReadOnly (0.14s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/Useradd (0.26s)
[2025-05-30T14:54:53.518Z] === RUN ostree.hotfix/rollback
[2025-05-30T14:55:15.379Z] --- PASS: ext.config.shared.ignition.remote (68.46s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix (131.57s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix/unlock (19.34s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix/install (1.01s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix/uninstall (0.58s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix/persist (33.57s)
[2025-05-30T14:55:33.411Z] --- PASS: ostree.hotfix/rollback (36.74s)
[2025-05-30T14:55:51.430Z] --- PASS: coreos.auth.verify (38.99s)
[2025-05-30T14:56:13.300Z] --- PASS: coreos.ignition.v2.users (40.33s)
[2025-05-30T14:56:16.560Z] --- PASS: coreos.ignition.failure (25.20s)
[2025-05-30T14:56:19.066Z] --- PASS: multipath.day2 (93.63s)
[2025-05-30T14:56:57.700Z] --- PASS: coreos.ignition.systemd.enable-service (39.70s)
[2025-05-30T14:57:07.623Z] --- PASS: ext.config.shared.networking.nmstate.policy (142.97s)
[2025-05-30T14:57:14.139Z] --- PASS: ext.config.shared.networking.mtu-on-bond-ignition (54.41s)
[2025-05-30T14:57:35.993Z] --- PASS: ext.config.shared.ignition.stable-boot (42.16s)
[2025-05-30T14:57:48.145Z] --- PASS: coreos.ignition.groups (39.44s)
[2025-05-30T14:57:50.649Z] === RUN rpmostree.upgrade-rollback/upgrade
[2025-05-30T14:58:29.279Z] --- PASS: ext.config.shared.networking.prefer-ignition-networking (48.79s)
[2025-05-30T14:58:29.279Z] --- PASS: ext.config.shared.ssh.custom-host-key-permissions (42.20s)
[2025-05-30T14:58:51.152Z] === RUN rpmostree.upgrade-rollback/rollback
[2025-05-30T14:58:52.512Z] --- PASS: kdump.crash.ssh (155.81s)
[2025-05-30T14:58:52.512Z] cluster.go:151: Running as unit: run-r1a8a98404bfa4f99ba5db0a15aceba72.service
[2025-05-30T14:59:14.374Z] --- PASS: ext.config.shared.networking.force-persist-ip (49.33s)
[2025-05-30T14:59:40.925Z] --- PASS: rpmostree.upgrade-rollback (147.12s)
[2025-05-30T14:59:40.925Z] --- PASS: rpmostree.upgrade-rollback/upgrade (60.35s)
[2025-05-30T14:59:40.925Z] --- PASS: rpmostree.upgrade-rollback/rollback (46.89s)
[2025-05-30T15:00:12.931Z] --- PASS: ext.config.shared.var-mount.luks (101.53s)
[2025-05-30T15:00:39.405Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.rpm-ostree.container-deps
[2025-05-30T15:00:39.405Z] --- PASS: fips.enable (105.63s)
[2025-05-30T15:00:39.405Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.license
[2025-05-30T15:00:39.964Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.growfs
[2025-05-30T15:00:40.887Z] === RUN non-exclusive-test-bucket-0/ext.config.files.localtime
[2025-05-30T15:00:41.811Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.check-symlink
[2025-05-30T15:00:42.369Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.systemd.no-systemd-firstboot
[2025-05-30T15:00:44.250Z] === RUN non-exclusive-test-bucket-0/ext.config.files.check-qemu-ga
[2025-05-30T15:00:45.172Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.validate-symlinks
[2025-05-30T15:00:59.991Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.root-immutable-bit
[2025-05-30T15:00:59.991Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.initrd.executables
[2025-05-30T15:01:03.255Z] === RUN non-exclusive-test-bucket-0/ext.config.files.conntrack-tools
[2025-05-30T15:01:03.815Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.sudoers
[2025-05-30T15:01:04.737Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.partition-scheme
[2025-05-30T15:01:06.618Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.nm-start
[2025-05-30T15:01:08.507Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.logrotate-service
[2025-05-30T15:01:09.867Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.rpmdb-sqlite
[2025-05-30T15:01:10.790Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.content-origins
[2025-05-30T15:01:12.671Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.aleph-version
[2025-05-30T15:01:13.231Z] === RUN non-exclusive-test-bucket-0/ext.config.files.env-godebug
[2025-05-30T15:01:16.485Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.no-google-device-links
[2025-05-30T15:01:17.045Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.composefs.enabled
[2025-05-30T15:01:18.928Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.podman.rootless-pasta-networking
[2025-05-30T15:01:25.454Z] --- PASS: ext.config.shared.reboot (74.49s)
[2025-05-30T15:01:27.974Z] === RUN non-exclusive-test-bucket-0/ext.config.files.composefs-maybe
[2025-05-30T15:01:28.228Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.podman.dns
[2025-05-30T15:01:31.495Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.root-prjquota
[2025-05-30T15:01:32.056Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.network-online-service
[2025-05-30T15:01:32.614Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.ignition.journald-log
[2025-05-30T15:01:34.497Z] --- PASS: ext.config.shared.ignition.delete-config (141.08s)
[2025-05-30T15:01:34.497Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.no-default-initramfs-net-propagation.default
[2025-05-30T15:01:35.419Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.setuid
[2025-05-30T15:01:37.301Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.nm-dhcp-client
[2025-05-30T15:01:37.861Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.yum-repo-dir
[2025-05-30T15:01:38.785Z] === RUN non-exclusive-test-bucket-0/ext.config.afterburn.sshkey-service
[2025-05-30T15:01:39.345Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.root-boot-ro
[2025-05-30T15:01:40.270Z] === RUN non-exclusive-test-bucket-0/ext.config.console.console-login-helper-service
[2025-05-30T15:01:41.201Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.systemd-repart-service
[2025-05-30T15:01:41.761Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.logging.printk
[2025-05-30T15:01:42.684Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.default-network-behavior-change
[2025-05-30T15:01:44.567Z] === RUN non-exclusive-test-bucket-0/ext.config.ssh.config
[2025-05-30T15:01:46.447Z] === RUN non-exclusive-test-bucket-0/ext.config.version.rhel-major-version
[2025-05-30T15:01:56.372Z] --- PASS: coreos.unique.boot.ignition.failure (21.50s)
[2025-05-30T15:02:00.528Z] --- PASS: ext.config.shared.podman.rootless-systemd (82.57s)
[2025-05-30T15:02:02.410Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.ntp.chrony.coreos-platform-chrony-config
[2025-05-30T15:02:02.969Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.clhm.network-device-info
[2025-05-30T15:02:03.893Z] === RUN non-exclusive-test-bucket-0/ext.config.version.drop-azure-ptp
[2025-05-30T15:02:04.451Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.selinux.usrlocal-context
[2025-05-30T15:02:05.374Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.systemd.default-unit-timeouts
[2025-05-30T15:02:07.257Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.console-config
[2025-05-30T15:02:08.624Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.tls
[2025-05-30T15:02:15.158Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.tmpfs
[2025-05-30T15:02:15.412Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.kubernetes.systemd-env-read
[2025-05-30T15:02:17.292Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.etc-permissions
[2025-05-30T15:02:19.173Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.file-directory-permissions
[2025-05-30T15:02:20.531Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.selinux.unmodified-policy
[2025-05-30T15:02:25.763Z] === RUN non-exclusive-test-bucket-0/ext.config.files.kata-osbuilder
[2025-05-30T15:02:26.322Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.setgid
[2025-05-30T15:02:28.203Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.selinux.podman-tmpfs-context
[2025-05-30T15:02:36.276Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.selinux.file-context-policy-match
[2025-05-30T15:02:38.157Z] --- PASS: ext.config.shared.var-mount.simple (73.45s)
[2025-05-30T15:02:50.312Z] --- PASS: ext.config.shared.ignition.kargs (54.48s)
[2025-05-30T15:03:05.133Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.nm-ifcfg-rh-plugin
[2025-05-30T15:03:06.493Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.root-bash
[2025-05-30T15:03:07.415Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.firewall.iptables
[2025-05-30T15:03:08.337Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.nic-naming
[2025-05-30T15:03:10.217Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.security.coreos-update-ca-trust
[2025-05-30T15:03:12.096Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.boot.bootupd
[2025-05-30T15:03:13.979Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.kernel-headers
[2025-05-30T15:03:14.537Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.initrd.compression
[2025-05-30T15:03:15.463Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.kubernetes.kube-watch
[2025-05-30T15:03:17.972Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.system-generators
[2025-05-30T15:03:19.330Z] --- PASS: ext.config.shared.ignition.systemd-enable-units (41.43s)
[2025-05-30T15:03:19.887Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.kdump.service
[2025-05-30T15:03:20.811Z] --- PASS: rhcos.selinux.manage (80.02s)
[2025-05-30T15:03:20.811Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.swap.zram-default
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0 (224.07s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.rpm-ostree.container-deps (1.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.license (0.74s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.growfs (0.82s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.localtime (0.75s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.check-symlink (0.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.systemd.no-systemd-firstboot (1.85s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.check-qemu-ga (0.77s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.validate-symlinks (12.47s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.root-immutable-bit (0.76s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.initrd.executables (4.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.conntrack-tools (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.sudoers (0.75s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.partition-scheme (1.78s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nm-start (1.77s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.logrotate-service (1.77s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.rpmdb-sqlite (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.content-origins (1.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.aleph-version (0.76s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.env-godebug (2.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.no-google-device-links (0.78s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.composefs.enabled (1.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.podman.rootless-pasta-networking (8.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.composefs-maybe (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.podman.dns (2.82s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.root-prjquota (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.network-online-service (0.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.ignition.journald-log (1.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.no-default-initramfs-net-propagation.default (0.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.setuid (1.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nm-dhcp-client (0.78s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.yum-repo-dir (0.77s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.afterburn.sshkey-service (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.root-boot-ro (0.76s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.console.console-login-helper-service (0.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.systemd-repart-service (0.79s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.logging.printk (0.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.default-network-behavior-change (1.85s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.ssh.config (1.82s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.version.rhel-major-version (15.93s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.ntp.chrony.coreos-platform-chrony-config (0.78s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.clhm.network-device-info (0.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.version.drop-azure-ptp (0.75s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.usrlocal-context (0.74s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.systemd.default-unit-timeouts (1.78s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.console-config (1.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.tls (5.80s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.tmpfs (0.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.kubernetes.systemd-env-read (1.78s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.etc-permissions (1.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.file-directory-permissions (1.79s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.unmodified-policy (4.79s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.kata-osbuilder (0.77s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.setgid (1.78s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.podman-tmpfs-context (7.83s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.file-context-policy-match (28.82s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nm-ifcfg-rh-plugin (1.86s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.root-bash (0.79s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.firewall.iptables (0.84s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nic-naming (1.85s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.security.coreos-update-ca-trust (1.82s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.boot.bootupd (1.81s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.kernel-headers (0.83s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.initrd.compression (0.81s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.kubernetes.kube-watch (2.86s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.system-generators (1.81s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.kdump.service (0.83s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.swap.zram-default (0.90s)
[2025-05-30T15:03:33.032Z] --- PASS: ext.config.shared.gshadow (42.37s)
[2025-05-30T15:04:11.662Z] --- PASS: ext.config.shared.networking.no-persist-ip (48.25s)
[2025-05-30T15:04:13.543Z] --- PASS: rhcos.sssd (40.81s)
[2025-05-30T15:04:52.177Z] --- PASS: ext.config.shared.ntp.chrony.dhcp-propagation (86.90s)
[2025-05-30T15:04:54.683Z] --- PASS: multipath.day1 (95.28s)
[2025-05-30T15:05:41.270Z] --- PASS: coreos.ignition.sethostname (39.65s)
[2025-05-30T15:05:42.628Z] --- PASS: ext.config.shared.var-mount.scsi-id (51.02s)
[2025-05-30T15:05:49.154Z] --- PASS: ext.config.shared.networking.ifname-karg.everyboot-systemd-link-file (95.74s)
[2025-05-30T15:05:59.077Z] --- PASS: fips.enable.partitions (108.57s)
[2025-05-30T15:06:31.074Z] --- PASS: ext.config.shared.networking.bridge-static-via-kargs (48.44s)
[2025-05-30T15:06:33.583Z] === RUN podman.base/info
[2025-05-30T15:06:34.505Z] === RUN podman.base/resources
[2025-05-30T15:06:49.331Z] --- PASS: podman.base (50.21s)
[2025-05-30T15:06:49.331Z] --- PASS: podman.base/info (1.07s)
[2025-05-30T15:06:49.331Z] --- PASS: podman.base/resources (9.93s)
[2025-05-30T15:06:49.331Z] --- PASS: coreos.ignition.mount.partitions (73.95s)
[2025-05-30T15:06:50.254Z] --- FAIL: kdump.crash.nfs (2.02s)
[2025-05-30T15:06:50.254Z] kdump.go:291: listen tcp :2049: bind: address already in use
[2025-05-30T15:07:05.075Z] --- PASS: rhcos.selinux.boolean.persist (75.01s)
[2025-05-30T15:07:26.941Z] --- PASS: ext.config.systemd.journal-compat (52.94s)
[2025-05-30T15:07:41.760Z] === RUN ostree.unlock/unlock
[2025-05-30T15:07:42.016Z] === RUN ostree.unlock/install
[2025-05-30T15:07:43.376Z] === RUN ostree.unlock/uninstall
[2025-05-30T15:07:43.934Z] === RUN ostree.unlock/discard
[2025-05-30T15:08:01.956Z] === RUN rhcos.services-disabled/ServicesDisabled
[2025-05-30T15:08:03.840Z] --- PASS: multipath.partition (73.31s)
[2025-05-30T15:08:04.398Z] --- PASS: rhcos.services-disabled (40.27s)
[2025-05-30T15:08:04.398Z] --- PASS: rhcos.services-disabled/ServicesDisabled (0.40s)
[2025-05-30T15:08:22.421Z] --- PASS: ostree.unlock (76.70s)
[2025-05-30T15:08:22.421Z] --- PASS: ostree.unlock/unlock (1.03s)
[2025-05-30T15:08:22.422Z] --- PASS: ostree.unlock/install (1.34s)
[2025-05-30T15:08:22.422Z] --- PASS: ostree.unlock/uninstall (0.67s)
[2025-05-30T15:08:22.422Z] --- PASS: ostree.unlock/discard (33.76s)
[2025-05-30T15:08:24.302Z] --- PASS: rhcos.network.multiple-nics (96.31s)
[2025-05-30T15:08:46.163Z] --- PASS: coreos.ignition.symlink (39.76s)
[2025-05-30T15:08:52.687Z] --- PASS: ext.config.shared.networking.kargs-rd-net (48.23s)
[2025-05-30T15:09:14.561Z] --- PASS: ext.config.shared.networking.no-default-initramfs-net-propagation.bootif (50.21s)
[2025-05-30T15:09:32.584Z] --- PASS: coreos.ignition.once (71.58s)
[2025-05-30T15:09:59.059Z] --- PASS: ext.config.shared.boot.grub2-install (43.53s)
[2025-05-30T15:09:59.059Z] --- PASS: coreos.misc.disk.varlibcontainers (75.83s)
[2025-05-30T15:09:59.059Z] cluster.go:151: Created symlink /etc/systemd/system/var-lib-containers.mount.wants/[email protected] → /etc/systemd/system/[email protected].
[2025-05-30T15:09:59.059Z] cluster.go:151: Created symlink /etc/systemd/system/local-fs.target.wants/var-lib-containers.mount → /etc/systemd/system/var-lib-containers.mount.
[2025-05-30T15:10:55.197Z] --- PASS: ext.config.shared.networking.nameserver (48.43s)
[2025-05-30T15:10:55.197Z] --- PASS: ext.config.shared.networking.nmstate.state (75.73s)
[2025-05-30T15:11:03.279Z] --- PASS: ext.config.shared.systemd.condition-needs-update (130.35s)
[2025-05-30T15:11:13.206Z] --- PASS: coreos.ignition.mount.disks (72.43s)
[2025-05-30T15:11:19.736Z] === RUN rpmostree.install-uninstall/install
[2025-05-30T15:11:29.662Z] --- PASS: ext.config.shared.clhm.ignition-warnings (40.83s)
[2025-05-30T15:11:56.145Z] --- PASS: ext.config.shared.containers.quadlet (44.55s)
[2025-05-30T15:12:08.297Z] --- PASS: ext.config.shared.ignition.resource.remote (38.49s)
[2025-05-30T15:12:09.657Z] --- FAIL: ostree.sync (2.02s)
[2025-05-30T15:12:09.657Z] sync.go:117: listen tcp :2049: bind: address already in use
[2025-05-30T15:12:36.136Z] === RUN rpmostree.install-uninstall/uninstall
[2025-05-30T15:13:14.783Z] --- PASS: ext.config.shared.disks.lvmdevices (75.45s)
[2025-05-30T15:13:32.806Z] --- PASS: ext.config.shared.networking.ifname-karg.udev-rule-firstboot-propagation (80.97s)
[2025-05-30T15:13:32.806Z] --- PASS: ext.config.shared.kdump.crash (149.09s)
[2025-05-30T15:13:42.730Z] --- PASS: rpmostree.install-uninstall (176.30s)
[2025-05-30T15:13:42.730Z] --- PASS: rpmostree.install-uninstall/install (74.08s)
[2025-05-30T15:13:42.730Z] --- PASS: rpmostree.install-uninstall/uninstall (65.40s)
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/add
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/list
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/show-url
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/refs
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/summary
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/delete
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote (41.97s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/add (0.33s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/list (0.13s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/show-url (0.25s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/refs (0.23s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/summary (0.52s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/delete (0.51s)
[2025-05-30T15:14:22.406Z] --- PASS: ext.config.shared.networking.mtu-on-bond-kargs (49.03s)
[2025-05-30T15:14:25.663Z] --- PASS: ext.config.shared.selinux.enforcing (73.70s)
[2025-05-30T15:14:25.663Z] mke2fs 1.47.2 (1-Jan-2025)
[2025-05-30T15:14:30.896Z] --- PASS: ext.config.shared.networking.team-dhcp-via-ignition (48.24s)
[2025-05-30T15:14:45.724Z] --- PASS: coreos.unique.boot.failure (18.75s)
[2025-05-30T15:15:03.744Z] --- PASS: ext.config.disk.nvme-compat-symlink (41.07s)
[2025-05-30T15:15:30.215Z] --- PASS: ext.config.shared.boot.bootupd-validate (59.30s)
[2025-05-30T15:16:16.824Z] --- PASS: ext.config.shared.networking.rd-net-timeout-carrier (86.07s)
[2025-05-30T15:16:16.824Z] --- PASS: coreos.selinux.boolean (71.74s)
[2025-05-30T15:18:53.236Z] KVM: Failed to create TCE64 table for liobn 0x80000001
[2025-05-30T15:18:59.774Z] --- PASS: ext.config.rpm-ostree.replace-rt-kernel (287.08s)
[2025-05-30T15:18:59.774Z] FAIL, output in /home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola
[2025-05-30T15:18:59.774Z]
[2025-05-30T15:18:59.774Z]
[2025-05-30T15:18:59.774Z] ======== Re-running failed tests (flake detection) ========
[2025-05-30T15:18:59.774Z]
[2025-05-30T15:18:59.774Z] ⏭️ Skipping kola test pattern "ext.config.version.rhaos-pkgs-match-openshift":
[2025-05-30T15:18:59.774Z] 👉 https://issues.redhat.com/browse/OCPBUGS-42688
[2025-05-30T15:18:59.774Z] ⏭️ Skipping kola test pattern "ext.config.shared.multipath.resilient":
[2025-05-30T15:18:59.774Z] 👉 https://github.com/coreos/fedora-coreos-tracker/issues/1937
[2025-05-30T15:19:00.028Z] === RUN ostree.sync
[2025-05-30T15:19:00.028Z] === RUN kdump.crash.nfs
[2025-05-30T15:19:00.585Z] qemu-system-ppc64: -netdev user,id=eth0,hostfwd=tcp:127.0.0.1:36287-:22,hostfwd=tcp:127.0.0.1:2049-:2049,hostname=qemu0: Could not set up host forwarding rule 'tcp:127.0.0.1:2049-:2049'
[2025-05-30T15:19:32.587Z] --- FAIL: ostree.sync (31.54s)
[2025-05-30T15:19:32.587Z] sync.go:117: failed to establish qmp connection: dial unix /var/tmp/mantle-qemu2099425295/qmp-1748618340436644144.sock: connect: connection refused
[2025-05-30T15:21:38.968Z] --- PASS: kdump.crash.nfs (157.10s)
[2025-05-30T15:21:38.968Z] cluster.go:151: Running as unit: run-rbc26bb21f6424c438e84e7781f60c9e3.service
[2025-05-30T15:21:38.968Z] FAIL, output in /home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola/rerun
[2025-05-30T15:21:38.968Z] Error: harness: test suite failed
[2025-05-30T15:21:38.968Z] 2025-05-30T15:21:37Z cli: harness: test suite failed
[2025-05-30T15:21:38.968Z] failed to execute cmd-kola: exit status 1
[2025-05-30T15:19:00.585Z] qemu-system-ppc64: -netdev user,id=eth0,hostfwd=tcp:127.0.0.1:36287-:22,hostfwd=tcp:127.0.0.1:2049-:2049,hostname=qemu0: Could not set up host forwarding rule 'tcp:127.0.0.1:2049-:2049'
[2025-05-30T15:19:32.587Z] --- FAIL: ostree.sync (31.54s)
[2025-05-30T15:19:32.587Z] sync.go:117: failed to establish qmp connection: dial unix /var/tmp/mantle-qemu2099425295/qmp-1748618340436644144.sock: connect: connection refused
Basically for whatever reason the host clashed: Could not set up host forwarding rule 'tcp:127.0.0.1:2049-:2049'. I don't know if another kdump.crash.nfs test was running for another pipeline (or if they would even clash).
You can also reproduce something similar with cosa kola run kdump.crash.nfs --multiply 2 --parallel 2.
From https://github.com/coreos/rhel-coreos-config/issues/89#issuecomment-3489542729
Both ostree.sync and kdump.crash.nfs need to bind port 2049 to the host, if they are running at the same time, one will be failed with listen tcp :2049: bind: address already in use.
I wonder if the more correct approach here would be to just mark these various tests as conflicts.
I don't think retrying as is done in https://github.com/coreos/coreos-assembler/pull/4358 will be reliable because the retry at 12*5seconds = 1minute probably isn't long enough.
Note that I don't know if we've conflicts before so it might not work perfectly on first try or it might!
I wonder if the more correct approach here would be to just mark these various tests as conflicts.
This sounds good to me and makes sense, thank you for the pointer.
I don't think retrying as is done in #4358 will be reliable because the retry at
12*5seconds = 1minuteprobably isn't long enough.
Agree it is not the good workaround.
Note that I don't know if we've
conflictsbefore so it might not work perfectly on first try or it might!
I will give it a try.
If I add Conflicts: []string{"kdump.crash.nfs"} in ostree.sync, then it will skip test ostree.sync, maybe this is by design. Actually we just do not want them to be running at the same time, or easier way is to skip ostree.sync running on qemu.
[coreos-assembler]$ kola run --qemu-image scos-9.0.20251029-0-qemu.x86_64.qcow2 ostree.sync kdump.crash.nfs --parallel 2
=== RUN kdump.crash.nfs
--- PASS: kdump.crash.nfs (178.63s)
cluster.go:151: Running as unit: run-r32f8b9d404f44ad18c3c9ac4425a3c77.service
PASS, output in _kola_temp/qemu-2025-11-06-0149-7071
hmm. not sure. When I do the same thing I get:
[dustymabe@media fcos]$ cosa kola run --parallel=2 ostree.sync kdump.crash.nfs
COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS=-v /var/b/shared/code/github.com/coreos/coreos-assembler//bin/:/usr/local/bin/:ro -v /var/b/shared/code/github.com/coreos/coreos-assembler/:/src/:rw
COSA_BUILD_WITH_BUILDAH=1
COREOS_ASSEMBLER_GIT=/var/b/shared/code/github.com/coreos/coreos-assembler/
COREOS_ASSEMBLER_CONFIG_GIT=/var/b/shared/code/github.com/coreos/fedora-coreos-config/
COREOS_ASSEMBLER_CONTAINER_NAME=cosafcos
+ podman run --rm -ti --security-opt label=disable --privileged --userns keep-id:uid=1000,gid=1000 -v /var/b/shared/assembler/fcos:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosafcos -e COSA_BUIs
kola -p qemu run --parallel=2 ostree.sync kdump.crash.nfs --output-dir tmp/kola
panic: exclusive test cannot have non-empty conflicts entry
which I guess makes sense. I guess we only introduce conflicts to deal with non-exclusive tests that conflict with each other, so they couldn't all be run on the same VM.
I think it could make sense to extend this concept even to exclusive tests like ostree.sync and kdump.crash.nfs.
[dustymabe@media fcos]$ cosa kola run --parallel=2 ostree.sync kdump.crash.nfs COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS=-v /var/b/shared/code/github.com/coreos/coreos-assembler//bin/:/usr/local/bin/:ro -v /var/b/shared/code/github.com/coreos/coreos-assembler/:/src/:rw COSA_BUILD_WITH_BUILDAH=1 COREOS_ASSEMBLER_GIT=/var/b/shared/code/github.com/coreos/coreos-assembler/ COREOS_ASSEMBLER_CONFIG_GIT=/var/b/shared/code/github.com/coreos/fedora-coreos-config/ COREOS_ASSEMBLER_CONTAINER_NAME=cosafcos + podman run --rm -ti --security-opt label=disable --privileged --userns keep-id:uid=1000,gid=1000 -v /var/b/shared/assembler/fcos:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosafcos -e COSA_BUIs kola -p qemu run --parallel=2 ostree.sync kdump.crash.nfs --output-dir tmp/kola panic: exclusive test cannot have non-empty conflicts entry
You are correct, seems I do not run the updated code.
$ ./bin/kola run --qemu-image scos-9.0.20251029-0-qemu.x86_64.qcow2 ostree.sync kdump.crash.nfs --parallel 2
panic: exclusive test cannot have non-empty conflicts entry
which I guess makes sense. I guess we only introduce conflicts to deal with non-exclusive tests that conflict with each other, so they couldn't all be run on the same VM.
I think it could make sense to extend this concept even to exclusive tests like
ostree.syncandkdump.crash.nfs.
In this case, should be make sure these 2 in the same bucket that would avoid to run parallelly?
In this case, should be make sure these 2 in the same bucket that would avoid to run parallelly?
It would take some internal work in kola, which might be useful.
It's kind of a hack, but another option is to tag them with reprovision, which forces them to run serially in our current setup. See https://github.com/coreos/fedora-coreos-config/pull/2826
Copy from https://github.com/coreos/fedora-coreos-config/pull/2826#issuecomment-1919428885:
It's always been a bit awkward how we mentioned the reprovision tag in the kola() step. We should probably add a new serial tag instead to all those tests and use that in kola().
I'd be happy with that. Though I think it would be nicer if there was a way to have it happen internally in kola instead of having to do --parallel=4 --tag=!serial and then --tag=serial. tests with the
serialtag would run serially even when--parallel=4was specified
This looks sane, maybe can add this in cosa in longer term.
And for short term, add tag reprovision in ostree.sync. WDYT?
This looks sane, maybe can add this in cosa in longer term.
Yeah. The other option here is to have something like;
Locks: ["HostPort=2049"]
and then have both tests define that and then the test framework would take/wait for a lock before continuing.
Note: Locks doesn't exist today. We'd have to implement it.