TestStorage.testSwap is flaky on centos-10

[jelly]

Explain what happens

1. run TestStorageswap.testBasic -vst in a loop
2. notice it fails in ~ 10 retries.

Findings

The page should start swap but instead of shows a disk without any swap partition. mkswap -f did not run but if you ask udisks, you will notice IdType is empty.

[root@centos-10-127-0-0-2-2201 ~]# udisksctl info -b /dev/sda | grep IdType
    IdType:

You can swapon /dev/sda so the swap partition is perfectly fine. So this is udev or udisks:

[root@centos-10-127-0-0-2-2201 ~]# udevadm info -q all /dev/disk/by-diskseq/3
P: /devices/pci0000:00/0000:00:01.1/ata1/host1/target1:0:1/1:0:1:0/block/sr0
M: sr0
R: 0
U: block
T: disk
D: b 11:0
N: sr0
L: -100
S: disk/by-path/pci-0000:00:01.1-ata-1.1
S: disk/by-diskseq/3
S: cdrom
S: disk/by-path/pci-0000:00:01.1-ata-1
S: disk/by-label/cidata
S: disk/by-id/ata-QEMU_DVD-ROM_QM00002
S: disk/by-uuid/2022-10-20-15-47-48-30
Q: 3
E: DEVPATH=/devices/pci0000:00/0000:00:01.1/ata1/host1/target1:0:1/1:0:1:0/block/sr0
[root@centos-10-127-0-0-2-2201 ~]# udevadm info -q all /dev/disk/by-diskseq/4
P: /devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda
M: sda
U: block
T: disk
D: b 8:0
N: sda
L: 0
S: disk/by-id/scsi-33333333000001f40
S: disk/by-id/wwn-0x3333333000001f40
S: disk/by-diskseq/4
Q: 4
E: DEVPATH=/devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda
E: DEVNAME=/dev/sda
E: DEVTYPE=disk
E: DISKSEQ=4
E: MAJOR=8
E: MINOR=0
...skipping...
E: SCSI_TPGS=0
E: SCSI_TYPE=disk
E: SCSI_VENDOR=Linux
E: SCSI_VENDOR_ENC=Linux\x20\x20\x20
E: ID_SCSI_INQUIRY=1
E: SCSI_IDENT_SERIAL=8000
E: SCSI_IDENT_LUN_T10=Linux_scsi_debug_8000
E: SCSI_IDENT_LUN_NAA_LOCAL=3333333000001f40
E: SCSI_IDENT_PORT_RELATIVE=1
E: SCSI_IDENT_PORT_NAA_LOCAL=3222222000001f3e
E: SCSI_IDENT_PORT_TARGET_PORT_GROUP=0x400
E: SCSI_IDENT_TARGET_NAA_LOCAL=3222222000001f3d
E: SCSI_IDENT_TARGET_NAME=naa.3222222000001F3D
E: DEVLINKS=/dev/disk/by-id/scsi-33333333000001f40 /dev/disk/by-id/wwn-0x3333333000001f40 /dev/disk/by-diskseq/4
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:
P: /devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda
M: sda
U: block
T: disk
D: b 8:0
N: sda
L: 0
S: disk/by-id/scsi-33333333000001f40
S: disk/by-id/wwn-0x3333333000001f40
S: disk/by-diskseq/4
Q: 4
E: DEVPATH=/devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda
E: DEVNAME=/dev/sda
E: DEVTYPE=disk
E: DISKSEQ=4
E: MAJOR=8
E: MINOR=0
...skipping...
E: SCSI_TPGS=0
E: SCSI_TYPE=disk
E: SCSI_VENDOR=Linux
E: SCSI_VENDOR_ENC=Linux\x20\x20\x20
E: ID_SCSI_INQUIRY=1
E: SCSI_IDENT_SERIAL=8000
E: SCSI_IDENT_LUN_T10=Linux_scsi_debug_8000
E: SCSI_IDENT_LUN_NAA_LOCAL=3333333000001f40
E: SCSI_IDENT_PORT_RELATIVE=1
E: SCSI_IDENT_PORT_NAA_LOCAL=3222222000001f3e
E: SCSI_IDENT_PORT_TARGET_PORT_GROUP=0x400
E: SCSI_IDENT_TARGET_NAA_LOCAL=3222222000001f3d
E: SCSI_IDENT_TARGET_NAME=naa.3222222000001F3D
E: DEVLINKS=/dev/disk/by-id/scsi-33333333000001f40 /dev/disk/by-id/wwn-0x3333333000001f40 /dev/disk/by-diskseq/4
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:

Clearly shows it is udev not picking up the change, after udevadm trigger:

E: ID_FS_BLOCKSIZE=4096
E: ID_FS_SIZE=52424704
E: ID_FS_UUID=f1617ace-7e4d-4c96-8982-8f14b5561645
E: ID_FS_UUID_ENC=f1617ace-7e4d-4c96-8982-8f14b5561645
E: ID_FS_VERSION=1
E: ID_FS_TYPE=swap
E: ID_FS_USAGE=other

udev events:

monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[26.948864] remove   /devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda/sda1 (block)
KERNEL[26.951799] change   /devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda (block)
KERNEL[26.972330] change   /devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda (block)
KERNEL[26.973019] change   /devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda (block)
UDEV  [26.976039] remove   /devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda/sda1 (block)
UDEV  [27.047107] change   /devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda (block)
UDEV  [28.094341] change   /devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda (block)
UDEV  [28.212420] change   /devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda (block)

Calling mkswap -f again:

UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[507.160165] change   /devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda (block)
UDEV  [507.253559] change   /devices/pseudo_0/adapter0/host3/target3:0:0/3:0:0:0/block/sda (block)