Flatcar fails to boot with error: `BPF: Max chain length or cycle detected. failed to validate module [zstd_compress] BTF: -40`

[ader1990]

Description

When trying to do dev and test on packages from sys-kernel (like dracut, bootengine, coreos-kernel), after multiple rebuilds, the image is stuck on boot with the error failed to validate module [zstd_compress] BTF: -40.

This error never happens on a clean env, first build workflow.

More error logs:

[    0.891657] SCSI subsystem initialized
[    0.904977] Loading iSCSI transport class v2.0-870.
[    0.922794] iscsi: registered transport (tcp)
[    0.951762] iscsi: registered transport (qla4xxx)
[    0.952464] QLogic iSCSI HBA Driver
[    1.024075] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
[    1.026133] device-mapper: uevent: version 1.0.3
[    1.027314] device-mapper: ioctl: 4.48.0-ioctl (2023-03-01) initialised: [email protected]
[    1.066608] raid6: avx2x4   gen() 38743 MB/s
[    1.083619] raid6: avx2x2   gen() 43987 MB/s
[    1.100607] raid6: avx2x1   gen() 41128 MB/s
[    1.101062] raid6: using algorithm avx2x2 gen() 43987 MB/s
[    1.118608] raid6: .... xor() 30514 MB/s, rmw enabled
[    1.119138] raid6: using avx2x2 recovery algorithm
[    1.133000] BPF: Max chain length or cycle detected
[    1.133549] failed to validate module [zstd_compress] BTF: -40
[    1.279450] cryptd: max_cpu_qlen set to 1000
[    1.279472] virtio_blk virtio3: 14/0/0 default/read/poll queues
[    1.282560] virtio_blk virtio3: [vda] 17805312 512-byte logical blocks (9.12 GB/8.49 GiB)
[    1.288724] AVX2 version of gcm_enc/dec engaged.
[    1.289079]  vda: vda1 vda2 vda3 vda4 vda6 vda7 vda9
[    1.290899] AES CTR mode by8 optimization enabled
[    1.306599] ACPI: \_SB_.GSIA: Enabled at IRQ 16
[    1.307933] ahci 0000:00:1f.2: AHCI 0001.0000 32 slots 6 ports 1.5 Gbps 0x3f impl SATA mode
[    1.310198] ahci 0000:00:1f.2: flags: 64bit ncq only
[    1.312171] scsi host0: ahci
[    1.312884] scsi host1: ahci
[    1.313006] scsi host2: ahci
[    1.314125] scsi host3: ahci
[    1.314353] scsi host4: ahci
[    1.320719] scsi host5: ahci
[    1.320778] ata1: SATA max UDMA/133 abar m4096@0xfebd5000 port 0xfebd5100 irq 44
[    1.321879] ata2: SATA max UDMA/133 abar m4096@0xfebd5000 port 0xfebd5180 irq 44
[    1.321883] ata3: SATA max UDMA/133 abar m4096@0xfebd5000 port 0xfebd5200 irq 44
[    1.321887] ata4: SATA max UDMA/133 abar m4096@0xfebd5000 port 0xfebd5280 irq 44
[    1.321890] ata5: SATA max UDMA/133 abar m4096@0xfebd5000 port 0xfebd5300 irq 44
[    1.321895] ata6: SATA max UDMA/133 abar m4096@0xfebd5000 port 0xfebd5380 irq 44
[    1.330411] BPF: Max chain length or cycle detected
[    1.330975] failed to validate module [zstd_compress] BTF: -40
[    1.634246] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[    1.635229] ata5: SATA link down (SStatus 0 SControl 300)
[    1.636013] ata1: SATA link down (SStatus 0 SControl 300)
[    1.636901] ata2: SATA link down (SStatus 0 SControl 300)
[    1.637822] ata4: SATA link down (SStatus 0 SControl 300)
[    1.646177] ata6: SATA link down (SStatus 0 SControl 300)
[    1.647424] ata3.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
[    1.648796] ata3.00: applying bridge limits
[    1.649919] ata3.00: configured for UDMA/100
[    1.651782] scsi 2:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ PQ: 0 ANSI: 5
[    1.695918] sr 2:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray
[    1.697226] cdrom: Uniform CD-ROM driver Revision: 3.20

Impact

Flatcar built images cannot boot.

Environment and steps to reproduce

1. Clone flatcar/scripts, run_sdk_container, build_packages, build_image, image_to_vm, flatcar_production_qemu.sh All works as expected.
2. Change the bootengine commit id, emerge bootengine, emerge the coreos-kernel, build_packages, build_image, image_to_vm, flatcar_production_qemu.sh The instance is stuck in early booting (initrd) before switch root with the above error message.

I cannot reproduce this behaviour accurately, sometimes it happens if I change bootengine commit id several times or when changing the kernel config.