MicroK8S (latest as well as 1.23/stable) is failed with "cni plugin not initialized"

[vishwanathjadhav]

Summary

I have installed MicroK8S version 1.23/stable, and after the installation checked the microk8s status, it is not running. This particular issue has been observed with MicroK8S versions higher than 1.18/stable.

What Should Happen Instead?

All the services and microk8s should be running, with the calico network, vxlan.calico virt interface is absent.

Reproduction Steps

1. Install the MicroK8S version > 1.18/stable
2. Check: microk8s status it will show it is not running. ifconfig -a will show "vxlan.calico" interface absent, also /var/log/syslog will be filled with the following error messages:
   • May 17 09:10:12 ip-10-31-2-247 microk8s.daemon-kubelite[12125]: E0517 09:10:12.024091 12125 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"

Introspection Report

inspection-report-20220517_102302.tar.gz

Can you suggest a fix?

Not Sure.

Are you interested in contributing with a fix?

@ktsakalozos, Please help me on the same.

[vishwanathjadhav]

I also see that my node status is NotReady.

[ktsakalozos]

Hi @vishwanathjadhav

Looking at the containerd logs (journalctl -u snap.microk8s.daemon-containerd) I see this error:

May 17 10:16:43 protegrity microk8s.daemon-containerd[3312]: time="2022-05-17T10:16:43.469324586Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:calico-node-q6xmn,Uid:b1ee2d93-4902-461b-a930-e13ec349ce67,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented: unknown"

Could you please go over the instructions in [1] and verify that cgroups are correctly setup?

[1] https://microk8s.io/docs/install-alternatives#heading--arm

[vishwanathjadhav]

@ktsakalozos - Thanks for your reply, I am able to fix this issue after adding the above changes along with the installation of conatinerd and docker. My Node is now ready state. But encountered a new issue, my deployed pods are not running properly. For additional information, we have a custom compiled kernel. Please let me know if there are some additional kernel dependencies present.

Cmd output:

:~# microk8s kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS              RESTARTS      AGE
kube-system   hostpath-provisioner-7764447d7c-27skk      0/1     ContainerCreating   0             3h59m
kube-system   coredns-64c6478b6c-6jbr5                   0/1     ContainerCreating   0             3h59m
kube-system   calico-kube-controllers-56999f44bb-7wm54   0/1     ContainerCreating   0             3h58m
kube-system   calico-node-pzx28                          0/1     Running             9 (56s ago)   20m

After a while

:~# microk8s kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS              RESTARTS      AGE
kube-system   hostpath-provisioner-7764447d7c-27skk      0/1     ContainerCreating   0             4h
kube-system   coredns-64c6478b6c-6jbr5                   0/1     ContainerCreating   0             4h
kube-system   calico-kube-controllers-56999f44bb-7wm54   0/1     ContainerCreating   0             4h
kube-system   calico-node-pzx28                          0/1     CrashLoopBackOff    9 (60s ago)   21m

inspection-report-20220524_161632.tar.gz

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.