microk8s

microk8s copied to clipboard

Summary

After an upgrade to 1.33 the cluster starts, but quickly stops working again. Looking through the logs, kubelite panics.

Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: E0426 16:05:23.053843   71239 runtime.go:142] "Observed a panic" panic="runtime error: invalid memory address or nil pointer dereference" panicGoValue="\"invalid memory address or nil pointer dereference\"" stacktrace=<
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         goroutine 7114 [running]:
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/apimachinery/pkg/util/runtime.logPanic({0x4f07580, 0x77ed780}, {0x3f9f0c0, 0x77584b0})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/apimachinery/pkg/util/runtime/runtime.go:132 +0xbc
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x4f07580, 0x77ed780}, {0x3f9f0c0, 0x77584b0}, {0x77ed780, 0x0, 0x10000c00d7cb740?})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0x116
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x4eb08a0?})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/apimachinery/pkg/util/runtime/runtime.go:64 +0x105
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         panic({0x3f9f0c0?, 0x77584b0?})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 runtime/panic.go:792 +0x132
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).doPodResizeAction(0xc0088d2340, 0xc008d6bb08, {0x0, 0x0, {0xc00b72ec40, 0x40}, 0x2, 0x0, {0x0, 0x0, ...}, ...})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:820 +0x94f
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).SyncPod(0xc0088d2340, {0x4f079c0, 0xc00db24dd0}, 0xc008d6bb08, 0xc00a77e580, {0x77ed780, 0x0, 0x0}, 0xc00881bb00)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:1409 +0x3de8
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet.(*Kubelet).SyncPod(0xc0086f1808, {0x4f07aa0, 0xc0078ad4f0}, 0x2, 0xc008d6bb08, 0x0, 0xc00a77e580)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/kubelet.go:2068 +0x1973
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop.func1(...)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1286
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop(0xc0088dd040, {0xc008291140, 0x24}, 0xc00a88d180)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1291 +0x57e
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod.func1()
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/pod_workers.go:950 +0x118
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         created by k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod in goroutine 5419
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:                 k8s.io/kubernetes/pkg/kubelet/pod_workers.go:945 +0x20db
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:  >
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: panic: runtime error: invalid memory address or nil pointer dereference [recovered]
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         panic: runtime error: invalid memory address or nil pointer dereference
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x398a9cf]
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: goroutine 7114 [running]:
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x4f07580, 0x77ed780}, {0x3f9f0c0, 0x77584b0}, {0x77ed780, 0x0, 0x10000c00d7cb740?})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/apimachinery/pkg/util/runtime/runtime.go:114 +0x1a9
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x4eb08a0?})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/apimachinery/pkg/util/runtime/runtime.go:64 +0x105
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: panic({0x3f9f0c0?, 0x77584b0?})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         runtime/panic.go:792 +0x132
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).doPodResizeAction(0xc0088d2340, 0xc008d6bb08, {0x0, 0x0, {0xc00b72ec40, 0x40}, 0x2, 0x0, {0x0, 0x0, ...}, ...})
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:820 +0x94f
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).SyncPod(0xc0088d2340, {0x4f079c0, 0xc00db24dd0}, 0xc008d6bb08, 0xc00a77e580, {0x77ed780, 0x0, 0x0}, 0xc00881bb00)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:1409 +0x3de8
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).SyncPod(0xc0086f1808, {0x4f07aa0, 0xc0078ad4f0}, 0x2, 0xc008d6bb08, 0x0, 0xc00a77e580)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/kubelet.go:2068 +0x1973
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop.func1(...)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1286
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop(0xc0088dd040, {0xc008291140, 0x24}, 0xc00a88d180)
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1291 +0x57e
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod.func1()
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:950 +0x118
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]: created by k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod in goroutine 5419
Apr 26 16:05:23 bernd microk8s.daemon-kubelite[71239]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:945 +0x20db

What Should Happen Instead?

Upgrading from 1.32 to 1.33 should work.

Reproduction Steps

1. Run 1.32(.3)/stable
2. Upgrade to 1.33/stable using sudo snap refresh microk8s --channel=1.33/stable

Introspection Report

inspection-report-20250426_161309.tar.gz

Can you suggest a fix?

No.

Are you interested in contributing with a fix?

No (I don't know enough about the issue).