[Bug] - Upgrade to 2023.7.20250331 fails if any DKMS module is installed

[jtprof]

Describe the bug Upgrade from any version to 2023.7.20250331 fails with kernel panic after reboot when any DKMS module is installed inside upgraded system

To Reproduce Steps to reproduce the behavior:

1. uname -r
2. dkms status
3. dnf upgrade --releasever=2023.7.20250331
4. reboot

Screenshots

Additional context It looks like the trouble cause in kernel-headers and kernel-devel packages version which doesn't much with kernel version which upgrade really installs.

[zcobol]

The kernel headers mismatch is present in the 2023.7.20250331 AMI also.

Launch the latest al2023 image. List kernel packages:

kernel-livepatch-repo-s3-2023.7.20250331-0.amzn2023.noarch
kernel-srpm-macros-1.0-14.amzn2023.0.2.noarch
kernel-6.1.131-143.221.amzn2023.aarch64
kernel-libbpf-6.12.20-23.97.amzn2023.aarch64
kernel-tools-6.12.20-23.97.amzn2023.aarch64
kernel-headers-6.12.20-23.97.amzn2023.aarch64
kernel-devel-6.12.20-23.97.amzn2023.aarch64

And dkms output when trying to build a driver:

ec2-user [ /usr/src/zfs-2.3.1 ]$ sudo dkms build -m zfs -v 2.3.1
Error! Your kernel headers for kernel 6.1.131-143.221.amzn2023.aarch64 cannot be found at /lib/modules/6.1.131-143.221.amzn2023.aarch64/build or /lib/modules/6.1.131-143.221.amzn2023.aarch64/source.
Please install the linux-headers-6.1.131-143.221.amzn2023.aarch64 package or use the --kernelsourcedir option to tell DKMS where it's located.

build and source are broken links:

ec2-user [ /usr/lib/modules/6.1.131-143.221.amzn2023.aarch64 ]$ file build
build: broken symbolic link to /usr/src/kernels/6.1.131-143.221.amzn2023.aarch64
ec2-user [ /usr/lib/modules/6.1.131-143.221.amzn2023.aarch64 ]$ file source
source: broken symbolic link to build

[bjoernd]

Thanks for reporting this. We are aware of the issue and working on a fixup.

As a suboptimal workaround for now, you can dnf install kernel-devel-$(uname -r) manually before installing the dkms package.

[jtprof]

kernel-devel-6.1.131-143.221.amzn2023.aarch64 is unavailable before upgrade to 2023.7.20250331. so the correct command sequence for my case is:

dnf remove zfs zfs-dkms dnf upgrade --releasever=2023.7.20250414 reboot dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r) dnf install -y zfd-dkms zfs zfs-dracut zfs_exporter python3-pyzfs zfs-debuginfo reboot

[bjoernd]

This should be fixed with the 2023.8.20250721 release.