Cannot get gVisor working on minikube

[david-mohr]

Description

I have tried different drivers (docker, kvm2, qemu) and different CNIs (bridge, kindnet, calico), but I can't seem to get the gvisor addon to work. Using the docker driver, I get permission denied errors (see https://stackoverflow.com/questions/72555998/cant-launch-pod-using-gvisor). I tried switching CNIs without success.

So I tried switching to a VM based solution (kvm2/qemu) but now once I enable the gVisor addon, the system goes into a crash loop (similar to https://github.com/google/gvisor/issues/5740). The pod description gives the same error on each pod:

Normal   SandboxChanged  5m19s (x4 over 7m26s)   kubelet            Pod sandbox changed, it will be killed and re-created.
Warning  BackOff         2m19s (x21 over 6m51s)  kubelet            Back-off restarting failed container

Steps to reproduce

I followed the instructions here: https://github.com/kubernetes/minikube/blob/master/deploy/addons/gvisor/README.md

minikube start --container-runtime=containerd --docker-opt containerd=/var/run/containerd/containerd.sock --driver=kvm2
minikube addons enable gvisor

$ kubectl get pod,runtimeclass gvisor -n kube-system
NAME         READY   STATUS             RESTARTS        AGE
pod/gvisor   0/1     CrashLoopBackOff   6 (3m15s ago)   11m

NAME                              HANDLER   AGE
runtimeclass.node.k8s.io/gvisor   runsc     11m

runsc version

runsc version release-20200211.0-16-gb2e86906ea4f
spec: 1.0.1-dev

docker version (if using docker)

# crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  1.6.6
RuntimeApiVersion:  v1alpha2

uname

Linux euclid 5.15.59-1-MANJARO #1 SMP PREEMPT Wed Aug 3 11:20:04 UTC 2022 x86_64 GNU/Linux

kubectl (if using Kubernetes)

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:30:46Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:18:48Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}


NAME      STATUS   ROLES           AGE     VERSION
minikube  Ready    control-plane   2m48s   v1.24.1

repo state (if built from source)

No response

runsc debug logs (if available)

Happy to provide these, just unsure how to extract them from minikube

[fvoznika]

containerd has updated its configuration format (once again). Minikube breaks because the plugin is trying to use the old format. Let me do a quick fix for now...we'll need a better way to patch config.toml to configure the runtime. Right now, it replaces the entire file and may lose other configuration changes.