Linux 6.7 super_block has s_shrink failing

[ptr1337]

Kernel: 6.7 rc1 Operating System: CachyOS (Archlinux)

Relevant Log:

checking whether sops->dirty_inode() wants flags... yes
checking whether super_block has s_shrink... configure: error: 
        *** None of the expected "sb->s_shrink()" interfaces were detected.
        *** This may be because your kernel version is newer than what is
        *** supported, or you are using a patched custom kernel with
        *** incompatible modifications.
        ***
        *** ZFS Version: zfs-2.2.99-174_g043c6ee3b
        *** Compatible Kernels: 3.10 - 6.5

[satmandu]

This might be due to the lockless slab shrink work?

e.g. https://lore.kernel.org/linux-erofs/[email protected]/ https://lwn.net/Articles/940644/

[ascendbeing]

I believe it can be narrowed down to this commit: https://github.com/torvalds/linux/commit/ecae0bd5173b1014f95a14a8dfbe40ec10367dcf it looks like they changed the s_shrink format in this commit to something zfs does not currently anticipate. so maybe try building against the commit just prior to this one, or maybe someone can do a patch/PR I was looking at https://github.com/torvalds/linux/commits/master/include/linux/fs.h to find this.

[ckane]

Thanks for all the details, I saw this too on my Arch system when trying to build the new kernel, but haven't had a chance to dig in.

[spikerguy]

Any update on this ?

I tried to build on aarch64 with upstream kernel 6.7-rc3 and failed with same error.

Thanks.

[setar]

arm64 , kernel 6.7-rc5 same error. I can't downgrade the kernel version as it doesn't have dev board orange5+ support. And now ZFS not work on this platform...

[DominoTree]

Just hit this with a regular Fedora update (building via DKMS/akmods) - I suspect others will be hitting it soon as well

[satmandu]

Wait for OpenZFS 2.2.3 to come out...

[gpmidi]

For the Fedora 39 users out there that need to roll back the kernel to get around this...

dnf install kernel-\*-6.5.6
dnf (re)install zfs zfs-dkms

[DominoTree]

For the Fedora 39 users out there that need to roll back the kernel to get around this...

dnf install kernel-\*-6.5.6
dnf (re)install zfs zfs-dkms

On F39, I did the following to get a list of installed kernels (I believe Fedora keeps the most recent 3) and switch back to an existing one prior to 6.7.x (6.6.14 happened to be the most recent)

I think the next kernel update will automatically go back to being the default, but I'm gonna keep an eye on it.

grubby --info=ALL
grubby --set-default=/boot/vmlinuz-6.6.14-200.fc39.x86_64

More info is here: https://www.golinuxcloud.com/grubby-command-examples/#6_List_installed_kernels

[edgan]

I took the 6.7 commits from this PR, https://github.com/openzfs/zfs/pull/15860, and with a little tweaking made a usable patch for ZFS 2.2.2 and 6.7.x Linux kernels. I have successfully used it to compile modules for 6.7.3 and 6.7.4. I am currently running it with 6.7.4 on Fedora 39.

Use at your own risk.

Steps:

cd /usr/src
wget https://cygnusx-1.org/zfs-2.2.2-6.7.patch
cd zfs-2.2.2
cat ../zfs-2.2.2-6.7.patch | patch -p1
dkms build -m zfs -v 2.2.2 -k 6.7.3-200.fc39.x86_64
dkms install -m zfs -v 2.2.2 -k 6.7.3-200.fc39.x86_64

[tleydxdy]

I wonder why can't the zfs fedora package block the newer kernels... should the dep for zfs be kernel-devel-matched instead of kernel-devel since kernel-devel doesn't depend on the kernel?

$ dnf repoquery --requires zfs-dkms
Last metadata expiration check: 4:57:37 ago on Mon 12 Feb 2024 11:48:19 PM CST.
/bin/sh
diffutils
dkms >= 2.2.0.3
gcc
kernel-devel <= 6.6.999
kernel-devel >= 3.10
make
perl

[gpmidi]

@tleydxdy The older one is probably installed to meet deps; no guarantee about which runs though.

[tleydxdy]

right I mean it should block 6.7 since that is not in even in the supported range.

[AllKind]

echo 'zfs' > /etc/dnf/protected.d/zfs.conf does the trick

[DominoTree]

echo 'zfs' > /etc/dnf/protected.d/zfs.conf does the trick

This file will prevent DNF from uninstalling ZFS while doing other updates, but won't prevent DNF from upgrading to a kernel that makes the DKMS module fail to build - ask me how I know :P

I believe this is also a default part of the ZFS installation on Fedora.

DNF is only aware of the zfs-dkms metapackage being installed, and not whether the dkms code will actually build properly on the next boot via akmods with whatever kernel Grub is pointed at when that happens - lots of moving parts to deal with.

[tleydxdy]

exactly, the zfs packages should block kernels with unsupported major version from even installing, it still might break for minor version updates I suppose but it's a lot better than nothing

[YesThatGy]

Fedora 39 user here. Ran into this a few weeks ago, did an update and still not fixed.

"the zfs packages should block kernels with unsupported major version from even installing"

Can I +100 vote for this?

[DominoTree]

I wonder if it’s possible to trigger akmods pointed at the new kernel during the dnf update process, so that the transaction is automatically rolled back if the dkms module fails to build against the new kernel.If not, I think the zfs dkms module would have to be built against kernels as they hit staging (or similar) and the release zfs package metadata would have to be updated if a failure occurs.On Feb 20, 2024, at 9:43 PM, YesThatGy @.***> wrote: Fedora 39 user here. Ran into this a few weeks ago, did an update and still not fixed. "the zfs packages should block kernels with unsupported major version from even installing" Can I +100 vote for this?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[tleydxdy]

I think that is a bit overkill, at the most basic, say zfs-2.2.2 is released with "Linux: compatible with 3.10 - 6.6 kernels" then the package can simply be mark as incompatible with kernel <=3.8 and kernel >=6.7

[DominoTree]

I like that approach - I wonder if it's "proper" to do it that way in the Fedora world - may be worth looking at how the Ubuntu zfs dkms packages handle this (if they do), or Fedora dkms packages from other projects like VirtualBox, because all dkms packages will have similar issues (and maybe other projects even ran into this specific breakage)

On Feb 21, 2024, at 2:42 AM, tleydxdy @.***> wrote:

I think that is a bit overkill, at the most basic, say zfs-2.2.2 is released with "Linux: compatible with 3.10 - 6.6 kernels" then the package can simply be mark as incompatible with kernel <=3.8 and kernel >=6.7

— Reply to this email directly, view it on GitHub https://github.com/openzfs/zfs/issues/15518#issuecomment-1955712806, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJPVJQATG3QN2PWOUGV4ILYUVGGZAVCNFSM6AAAAAA7JE62AWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJVG4YTEOBQGY. You are receiving this because you commented.

[AllKind]

dkms does not provide any mechanism to talk to each possible packet manager out there (AFAIK). If a distro is packaging the zfs-dkms or kmods it is responsible for configuring the relevant 'Conflict' or whatever definitions for their package (.deb or .rpm for example). Now the packages built from source by OpenZFS would need to identify each distribution (i.e. fedora, ubuntu, etc), most likely in the ./configure process and pass that information to the package building scripts (for i.e. .deb or .rpm) and there depending on which distro is in play define the 'Conflict' with the correct package name for that distribution (i.e. linux-source, kernel-source, kernel-devel, etc.) See my comment: https://github.com/openzfs/zfs/issues/15759#issuecomment-1951420427 Nobody has done the work (yet). I bet the OpenZFS developers would more then welcome a Pull Request from somebody volunteering for that.

[robn]

FYI, 2.2.3 is released: https://github.com/openzfs/zfs/releases/tag/zfs-2.2.3

[DominoTree]

Currently traveling but I think I'll take a crack at it when I get back next week - seems like a fairly easy area to get involved without knowing a ton about ZFS internals :)

On Feb 21, 2024, at 10:39 PM, Mart Frauenlob @.***> wrote:

dkms does not provide any mechanism to talk to each possible packet manager out there (AFAIK). If a distro is packaging the zfs-dkms or kmods it is responsible for configuring the relevant 'Conflict' or whatever definitions for their package (.deb or .rpm for example). Now the packages built from source by OpenZFS would need to identify each distribution (i.e. fedora, ubuntu, etc), most likely in the ./configure process and pass that information to the package building scripts (for i.e. .deb or .rpm) and there depending on which distro is in play define the 'Conflict' with the correct package name for that distribution (i.e. linux-source, kernel-source, kernel-devel, etc.) See my comment: #15759 (comment) https://github.com/openzfs/zfs/issues/15759#issuecomment-1951420427 Nobody has done the work (yet). I bet the OpenZFS developers would more then welcome a Pull Request from somebody volunteering for that.

— Reply to this email directly, view it on GitHub https://github.com/openzfs/zfs/issues/15518#issuecomment-1958007038, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJPVJSJ5QC4LR6HOKH3LETYUZSRDAVCNFSM6AAAAAA7JE62AWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJYGAYDOMBTHA. You are receiving this because you commented.

[AllKind]

@DominoTree Maybe this will come in handy: https://fedoraproject.org/wiki/Packaging:Conflicts

[AllKind]

@DominoTree I didn't notice it before, but looked at the zfs-dkms.spec.in file now and I see the min/max version and a requires definition are already there:

Requires:       kernel-devel >= @ZFS_META_KVER_MIN@, kernel-devel <= @[email protected]
Requires(post): kernel-devel >= @ZFS_META_KVER_MIN@, kernel-devel <= @[email protected]

So either that is not sufficient for RPM (Do we need a Conflicts definition?), or something else in the chain is broken??

[ptr1337]

Ill close this bug, since it is fixed in upstream.

[tleydxdy]

@AllKind @ptr1337 yes, require is not enough, need to have conflict as well. that line is already there but me and a few others still get broken system from kernel upgrades. also kernel-devel is different from kernel, it's only the headers

[ptr1337]

@AllKind @ptr1337 yes, require is not enough, need to have conflict as well. that line is already there but me and a few others still get broken system from kernel upgrades. also kernel-devel is different from kernel, it's only the headers

I think this should be in a separate issue discussed since this is not a direct issue with the current 6.7 issue described above.

[poige]

Ill close this bug, since it is fixed in upstream.

@ptr1337 That's ambiguous — which upstream(?), if there're 2 involved projects (kernel and ZFS).

References would be handy.

checking whether super_block has s_shrink... configure: error:
        *** None of the expected "sb->s_shrink()" interfaces were detected.
[…skip…]
        *** ZFS Version: zfs-2.2.3-1
        *** Compatible Kernels: 3.10 - 6.7

— I'm facing this very issue with zfs-2.2.3-1 and 6.7.9

[AllKind]

@poige ptr1337 meant ZFS upstream. Namely by this pull request: https://github.com/openzfs/zfs/pull/15694 If it fails for you with kernel 6.7, you should open a new issue providing the details of your setup. You should also provide your config.log.