Bucketed test logs are sometimes mixed together

[jlebon]

I was investigating https://github.com/coreos/fedora-coreos-config/issues/1637 and while looking at the test logs at e.g. tmp/kola/non-exclusive-test-bucket-0/$uuid/ext.config.kubernetes.kube-watch.txt, I noticed that test logs were not correctly separated. E.g.:

Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + . /var/opt/kola/extdata-32/commonlib.sh
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + EXPECTED_INITRD_NETWORK_CFG1='[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: id=Wired Connection
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: autoconnect-retries=1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: multi-connect=3
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: permissions=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: mac-address-blacklist=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dns-search=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: required-timeout=20000
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: addr-gen-mode=eui64
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dns-search=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [proxy]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [user]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: org.freedesktop.NetworkManager.origin=nm-initrd-generator'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + EXPECTED_INITRD_NETWORK_CFG2='[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: id=Wired Connection
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: autoconnect-retries=1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: multi-connect=3
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: permissions=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: mac-address-blacklist=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dns-search=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: required-timeout=20000
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: addr-gen-mode=eui64
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dns-search=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [proxy]'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + EXPECTED_INITRD_NETWORK_CFG3='[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: id=Wired Connection
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: autoconnect-retries=1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: multi-connect=3
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: required-timeout=20000
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: addr-gen-mode=eui64
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [proxy]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [user]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: org.freedesktop.NetworkManager.origin=nm-initrd-generator'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + EXPECTED_REALROOT_NETWORK_CFG1='[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: id=Wired connection 1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: autoconnect-priority=-999
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: interface-name=xxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: permissions=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: timestamp=xxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: mac-address-blacklist=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dns-search=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: addr-gen-mode=stable-privacy
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dns-search=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [proxy]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [.nmmeta]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: nm-generated=true'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + EXPECTED_REALROOT_NETWORK_CFG2='[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: id=Wired connection 1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: autoconnect-priority=-999
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: interface-name=xxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: timestamp=xxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: addr-gen-mode=stable-privacy
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [proxy]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [.nmmeta]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: nm-generated=true'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + source /etc/os-release
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ NAME='Fedora Linux'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ VERSION='36.20220406.10.1 (CoreOS Prerelease)'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ ID=fedora
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ VERSION_ID=36
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ VERSION_CODENAME=
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ PLATFORM_ID=platform:f36
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ PRETTY_NAME='Fedora CoreOS 36.20220406.10.1'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ ANSI_COLOR='0;38;2;60;110;180'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ LOGO=fedora-logo-icon
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ CPE_NAME=cpe:/o:fedoraproject:fedora:36
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ HOME_URL=https://getfedora.org/coreos/
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ DOCUMENTATION_URL=https://docs.fedoraproject.org/en-US/fedora-coreos/
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ SUPPORT_URL=https://github.com/coreos/fedora-coreos-tracker/
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ BUG_REPORT_URL=https://github.com/coreos/fedora-coreos-tracker/
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ REDHAT_BUGZILLA_PRODUCT=Fedora
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ REDHAT_BUGZILLA_PRODUCT_VERSION=36
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ REDHAT_SUPPORT_PRODUCT=Fedora
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ REDHAT_SUPPORT_PRODUCT_VERSION=36
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ PRIVACY_POLICY_URL=https://fedoraproject.org/wiki/Legal:PrivacyPolicy
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ VARIANT=CoreOS
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ VARIANT_ID=coreos
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ OSTREE_VERSION=36.20220406.10.1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ DEFAULT_HOSTNAME=localhost
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + '[' fedora == fedora ']'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + '[' 36 -ge 36 ']'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + EXPECTED_INITRD_NETWORK_CFG='[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: id=Wired Connection
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: autoconnect-retries=1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: multi-connect=3
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: required-timeout=20000
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: addr-gen-mode=eui64
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [proxy]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [user]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: org.freedesktop.NetworkManager.origin=nm-initrd-generator'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + EXPECTED_REALROOT_NETWORK_CFG='[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: id=Wired connection 1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: autoconnect-priority=-999
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: interface-name=xxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: timestamp=xxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: addr-gen-mode=stable-privacy
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [proxy]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [.nmmeta]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: nm-generated=true'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + DEFAULT_KARGS_FILE=/usr/lib/dracut/modules.d/35coreos-network/50-afterburn-network-kargs-default.conf
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + source /dev/fd/63
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4074]: ++ grep -o 'AFTERBURN_NETWORK_KARGS_DEFAULT=.*' /usr/lib/dracut/modules.d/35coreos-network/50-afterburn-network-kargs-default.conf
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ++ AFTERBURN_NETWORK_KARGS_DEFAULT=ip=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4076]: ++ mktemp -d
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + tmpdir=/tmp/tmp.qojFwc0PJh
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + /usr/libexec/nm-initrd-generator -c /tmp/tmp.qojFwc0PJh/connections -i /tmp/tmp.qojFwc0PJh/initrd-data-dir -r /tmp/tmp.qojFwc0PJh/conf.d -- ip=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4078]: ++ normalize_connection_file /tmp/tmp.qojFwc0PJh/connections/default_connection.nmconnection
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4078]: ++ sed -e 's/^uuid=.*$/uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/' -e 's/^timestamp=.*$/timestamp=xxxxxxxxxx/' -e 's/^interface-name=.*$/interface-name=xxxx/' /tmp/tmp.qojFwc0PJh/connections/default_connection.nmconnection
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + GENERATED_INITRD_NETWORK_CFG='[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: id=Wired Connection
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: autoconnect-retries=1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: multi-connect=3
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: required-timeout=20000
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: addr-gen-mode=eui64
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [proxy]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [user]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: org.freedesktop.NetworkManager.origin=nm-initrd-generator'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + diff -u /dev/fd/63 /dev/fd/62
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: ++ echo '[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: id=Wired Connection
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: autoconnect-retries=1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: multi-connect=3
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: required-timeout=20000
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: addr-gen-mode=eui64
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: [proxy]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: [user]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4080]: org.freedesktop.NetworkManager.origin=nm-initrd-generator'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: ++ echo '[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: id=Wired Connection
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: autoconnect-retries=1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: multi-connect=3
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: required-timeout=20000
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: addr-gen-mode=eui64
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: dhcp-timeout=90
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: [proxy]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: [user]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4079]: org.freedesktop.NetworkManager.origin=nm-initrd-generator'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4082]: ++ normalize_connection_file /dev/fd/63
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4082]: ++ sed -e 's/^uuid=.*$/uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/' -e 's/^timestamp=.*$/timestamp=xxxxxxxxxx/' -e 's/^interface-name=.*$/interface-name=xxxx/' /dev/fd/63
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4083]: +++ sudo cat '/run/NetworkManager/system-connections/Wired connection 1.nmconnection'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + GENERATED_REALROOT_NETWORK_CFG='[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: id=Wired connection 1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: autoconnect-priority=-999
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: interface-name=xxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: timestamp=xxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: addr-gen-mode=stable-privacy
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [proxy]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: [.nmmeta]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: nm-generated=true'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + diff -u /dev/fd/63 /dev/fd/62
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: ++ echo '[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: id=Wired connection 1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: autoconnect-priority=-999
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: interface-name=xxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: timestamp=xxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: addr-gen-mode=stable-privacy
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: [proxy]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: [.nmmeta]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4097]: nm-generated=true'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: ++ echo '[connection]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: id=Wired connection 1
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: type=ethernet
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: autoconnect-priority=-999
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: interface-name=xxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: timestamp=xxxxxxxxxx
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: [ethernet]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: [ipv4]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: [ipv6]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: addr-gen-mode=stable-privacy
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: method=auto
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: [proxy]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: [.nmmeta]
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4096]: nm-generated=true'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + ok 'success: expected network configs were generated'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: + echo ok 'success: expected network configs were generated'
Apr 06 17:49:50 qemu0 kola-runext-test.sh[4073]: ok success: expected network configs were generated
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: + . /var/opt/kola/extdata-28/commonlib.sh
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4304]: ++ systemctl is-active kube-watch.path
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: + '[' active '!=' active ']'
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: + ok 'kube-watch.path successfully activated'
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: + echo ok 'kube-watch.path successfully activated'
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: ok kube-watch.path successfully activated
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: + touch /etc/kubernetes/kubeconfig
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: + ok 'successfully created /etc/kubernetes/kubeconfig'
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: + echo ok 'successfully created /etc/kubernetes/kubeconfig'
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: ok successfully created /etc/kubernetes/kubeconfig
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4306]: ++ systemctl is-active kube-watch.service
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: + '[' inactive == activating ']'
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4307]: ++ systemctl is-active kube-watch.service
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: + '[' inactive '!=' active ']'
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: + fatal 'kube-watch.service did not successfully activate'
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: + echo 'kube-watch.service did not successfully activate'
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: kube-watch.service did not successfully activate
Apr 06 17:49:52 qemu0 kola-runext-test.sh[4303]: + exit 1

The "ok successfully created /etc/kubernetes/kubeconfig" belongs to that test, but e.g. "ok success: expected network configs were generated" comes from ext.config.networking.default-network-behavior-change. And just looking at the pid in brackets, you can tell multiple tests are intermingled.

[dustymabe]

Are they running in parallel (i.e. kola run --parallel=5)?

[jlebon]

Yes, this is from the pipeline. E.g. copy/pasting logs from https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/build/454/consoleFull:

18:03:45  kola -p qemu-unpriv --output-dir tmp/kola run --rerun --parallel 6 --no-test-exit-error

[dustymabe]

Ahh, but because you were looking at the single test scoped ext.config.kubernetes.kube-watch.txt it shouldn't have any other test output in there. Got ya.

[jlebon]

Ran into this again. Not a big deal because I can look at the full journal.txt but it'd still be good to fix this.