memory.kmem.limit_in_bytes: operation not supported: unknown

[bhperry]

Running microk8s 1.21 on popOS 20.04. Ran a full system upgrade yesterday, which brough my kernel up to 5.16.11-76051611-generic. Any pods that have memory requests/limits stopped being able to start. They get events like:

  Warning  Failed     0s                 kubelet            Error: failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:326: applying cgroup configuration for process caused: failed to write 1 to memory.kmem.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod1b2f7320-b18c-4f85-ab3b-8184d1eb3974/coredns/memory.kmem.limit_in_bytes: operation not supported: unknown

Reinstalling microk8s 1.21 didn't fix the problem, but eventually I tried installing 1.22 and that did work. Would like to use 1.21 though, as we use microk8s as a dev env to mirror our production kubernetes clusters, which are running k8s 1.21.

Wondering if it could be an issue with the version of containerd installed in microk8s 1.21? I see that it was upgraded in the 1.22 release.

inspection-report-20220405_102016.tar.gz

[neoaggelos]

Yes, MicroK8s 1.21 is using containerd 1.4, which is EOL as of March 3rd, 2022.

[bhperry]

Ok, thanks for the info. No plans to update microk8s 1.21's version of containerd?

[AdamIsrael]

I'm running into the same issue:

microk8s 1.18.20 Ubuntu 20.04.4 Kernel 5.16.11-76051611-generic

For similar reasons to @bhperry, I can't upgrade the microk8s version.

This seems more like a kernel issue. I rebooted into kernel 5.15.15-76051515-generic and my deployment works as expected. That's at least a workaround: downgrade to a 5.15.x kernel to use microk8s 1.21 and below.

[bsilver8192]

For reference, this looks like an upstream kernel regression. https://discuss.hashicorp.com/t/exec-driver-memory-kmem-limit-in-bytes-operation-not-supported/35242 mentions the same problem. I'm seeing it with Debian 5.16.0-0.bpo.4-amd64 and Docker 20.10.5+dfsg1.

[robbat2]

Rebuild your docker & containerd after upgrading the Kernel.

[mfalkvidd]

@robbat2 are you aware of any instructions on how to rebuild docker & containerd? (when using mikcrok8s on Ubuntu 22.04.1 LTS if that matters)

[mausch]

These kernel parameters fixed the issue for me on Ubuntu 22.04.3 kernel 6.2.0-26 : https://doc.arvados.org/main/install/install-docker.html