Uninstalling Multus doesn't clean up files properly

[arches-richard]

Name and Version

bitnami/multus-cni 2.0.2

What architecture are you using?

amd64

What steps will reproduce the bug?

1. helm install -n multus multus oci://registry-1.docker.io/bitnamicharts/multus-cni
2. helm uninstall -n multus multus
3. Check .Values.hostCNIBinDir and .Values.hostCNINetDir on each node

What is the expected behavior?

All Multus-related files are deleted when the chart is uninstalled

What do you see instead?

With the values:

hostCNIBinDir: '/var/lib/rancher/k3s/data/current/bin'
hostCNINetDir: '/var/lib/rancher/k3s/agent/etc/cni/net.d'

I see these files even after the chart is uninstalled:

/var/lib/rancher/k3s/data/current/bin/
/var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.
/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d

Additional information

This causes a bunch of things to break when the chart is uninstalled. For instance, trying to install the cert-manager chart results in this:

kubectl events -n cert-manager pod/cert-manager-5f8646db6b-p8j2c

LAST SEEN                  TYPE      REASON                   OBJECT                                          MESSAGE
4m5s                       Warning   FailedCreatePodSandBox   Pod/cert-manager-webhook-687b7f8b97-hs9z8       Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b9a5aef12cf7520a3ed29ec4403b6a7dc99bfe7acfc43a5083ebe9a9f3b26472": plugin type="multus" failed (add): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory
4m5s                       Warning   FailedCreatePodSandBox   Pod/cert-manager-5f8646db6b-p8j2c               Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5360eab64782d166d368c56fcea5eb5f5981f6d93b11f1ddb426070afaee7f1b": plugin type="multus" failed (add): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory
4m5s                       Warning   FailedCreatePodSandBox   Pod/cert-manager-startupapicheck-phzbc          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "26fa24270031cde33912f45a50378da59c8e204d1d6d852b3ea3446a9cc620ad": plugin type="multus" failed (add): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory
4m5s                       Warning   FailedCreatePodSandBox   Pod/cert-manager-cainjector-5cf5f57dd7-lhqjg    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b925befe1b9e1478e29ea96a55339d08095cf5c4ddfdf910fbdec707fdfa95c9": plugin type="multus" failed (add): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory
3m16s (x5308 over 3h16m)   Warning   FailedKillPod            Pod/cert-manager-webhook-687b7f8b97-7wpw7       error killing pod: failed to "KillPodSandbox" for "e9aa6226-c620-4004-a6b6-f484ea6532c5" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"9af12021aa1b9c4a68ea0b150a81ac1f8a309ff0f946328ce266925bed1068a8\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
3m16s (x5308 over 3h16m)   Warning   FailedKillPod            Pod/cert-manager-5f8646db6b-v2kdh               error killing pod: failed to "KillPodSandbox" for "3aca411b-2e5c-4c0c-9db5-008a36cae112" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"50edd5a28dcb26c26ffd7a3544d97dea4a082243b54f281ee4fb8909cc00ad2c\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
3m16s (x5309 over 3h16m)   Warning   FailedKillPod            Pod/cert-manager-startupapicheck-btfhr          error killing pod: failed to "KillPodSandbox" for "ac8824ee-8228-47d9-b9e5-876e6eb205d8" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"5c83f6e19e2adefb0160559460449e30dd337f0fa91eabbdcb0a9df028af3331\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
3m12s (x471 over 31m)      Warning   FailedKillPod            Pod/cert-manager-cainjector-5cf5f57dd7-8jt2l    error killing pod: failed to "KillPodSandbox" for "5f0b8d21-cd79-4cf0-9eed-6fa02c01fa55" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"664d8232b37d332e924146b890bc9f069fc82dfd4f134902104c8ed15cad20dd\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
3m12s (x470 over 31m)      Warning   FailedKillPod            Pod/cert-manager-webhook-687b7f8b97-xx4ch       error killing pod: failed to "KillPodSandbox" for "089f653f-eedd-4999-a655-d7e5ac16d901" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"c6311e8cc319cf4e47d99e6d7d8416eaf48b4010fb8dd6f48ebe36a7fb3bae1d\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
3m12s (x471 over 31m)      Warning   FailedKillPod            Pod/cert-manager-5f8646db6b-7dvn6               error killing pod: failed to "KillPodSandbox" for "3f2d15cc-4f22-42f9-8dbd-8bab95ebdefa" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"370956fd6e19795f37098cb652019a16fd7c10fd60ce45eacb1f807c818bdfa9\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
3m12s (x422 over 20m)      Warning   FailedKillPod            Pod/cert-manager-startupapicheck-n579m          error killing pod: failed to "KillPodSandbox" for "9ad3586d-7177-4870-a892-3a95a4a00205" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"e5a982ee428918483160810f5d5dee3a3386d7d63f80ef156603ae633b3b025a\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
3m2s                       Normal    SuccessfulCreate         ReplicaSet/cert-manager-5f8646db6b              Created pod: cert-manager-5f8646db6b-p8j2c
3m2s                       Normal    ScalingReplicaSet        Deployment/cert-manager                         Scaled up replica set cert-manager-5f8646db6b to 1
3m2s                       Normal    ScalingReplicaSet        Deployment/cert-manager-webhook                 Scaled up replica set cert-manager-webhook-687b7f8b97 to 1
3m2s                       Normal    SuccessfulCreate         ReplicaSet/cert-manager-cainjector-5cf5f57dd7   Created pod: cert-manager-cainjector-5cf5f57dd7-lhqjg
3m2s                       Normal    ScalingReplicaSet        Deployment/cert-manager-cainjector              Scaled up replica set cert-manager-cainjector-5cf5f57dd7 to 1
3m2s                       Normal    SuccessfulCreate         ReplicaSet/cert-manager-webhook-687b7f8b97      Created pod: cert-manager-webhook-687b7f8b97-hs9z8
3m1s                       Normal    Scheduled                Pod/cert-manager-5f8646db6b-p8j2c               Successfully assigned cert-manager/cert-manager-5f8646db6b-p8j2c to tesla
3m1s                       Normal    Scheduled                Pod/cert-manager-cainjector-5cf5f57dd7-lhqjg    Successfully assigned cert-manager/cert-manager-cainjector-5cf5f57dd7-lhqjg to tesla
3m1s                       Normal    Scheduled                Pod/cert-manager-webhook-687b7f8b97-hs9z8       Successfully assigned cert-manager/cert-manager-webhook-687b7f8b97-hs9z8 to tesla
3m1s                       Normal    SuccessfulCreate         Job/cert-manager-startupapicheck                Created pod: cert-manager-startupapicheck-phzbc
3m                         Normal    Scheduled                Pod/cert-manager-startupapicheck-phzbc          Successfully assigned cert-manager/cert-manager-startupapicheck-phzbc to tesla
2m42s (x5429 over 3h25m)   Warning   FailedKillPod            Pod/cert-manager-5f8646db6b-5nq7x               error killing pod: failed to "KillPodSandbox" for "fc9fe7e0-5313-433b-a252-839e8ec1c95b" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"db6ece786d656784a9b22cb7194cdd9258a8f193e41c3a472c54f9fce0f62d16\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
2m30s (x4229 over 155m)    Warning   FailedKillPod            Pod/cert-manager-5f8646db6b-8vb4m               error killing pod: failed to "KillPodSandbox" for "4831e723-0700-4dd4-b4a1-9bd878ebd8ad" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"81f281b39c598b91d3a99559a17eacf5261e5adf745e6fad5a3ce4472ac10083\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
2m30s (x4229 over 155m)    Warning   FailedKillPod            Pod/cert-manager-cainjector-5cf5f57dd7-2rkvf    error killing pod: failed to "KillPodSandbox" for "d90cc1cd-c4b2-469a-928f-24946264dc9d" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"99f4bf07cdef1fb0107c3b14b1e998a8a8c0da7591d569ae2c437a8b3d126019\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
114s (x4219 over 149m)     Warning   FailedKillPod            Pod/cert-manager-webhook-687b7f8b97-qnb2p       error killing pod: failed to "KillPodSandbox" for "113fc5b8-82ee-47b1-a25a-e62213f97625" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"1c199b708f6fb5484747a6abbdf9bbd25abac80de87222ec32e1e88b5d33adfc\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
114s (x4220 over 149m)     Warning   FailedKillPod            Pod/cert-manager-5f8646db6b-4lmj2               error killing pod: failed to "KillPodSandbox" for "362608f7-b636-4c10-96cd-c6020d4ea8cb" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"a6c6b62ee8a5890f43edd1bbfe0720c57056667ad6206ff0e6641fb5ed30ea6f\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
114s (x4220 over 149m)     Warning   FailedKillPod            Pod/cert-manager-cainjector-5cf5f57dd7-tjvff    error killing pod: failed to "KillPodSandbox" for "ad11a091-dba5-4efc-8ae7-9626a0845689" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"99feed413495b1c0bb5cb6ed773da5e870f9cb15dc3b83ebc45d0151fac13733\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
80s (x5475 over 3h40m)     Warning   FailedKillPod            Pod/cert-manager-webhook-687b7f8b97-jz69n       error killing pod: failed to "KillPodSandbox" for "e2fc1347-c758-4917-8f77-931cc0282271" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"9ed9e14fb35f33e1c0768e1c690fe0d1fd93846587985a3cdb4ed931c6591bc6\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
74s (x14 over 4m5s)        Normal    SandboxChanged           Pod/cert-manager-webhook-687b7f8b97-hs9z8       Pod sandbox changed, it will be killed and re-created.
72s (x14 over 4m4s)        Normal    SandboxChanged           Pod/cert-manager-cainjector-5cf5f57dd7-lhqjg    Pod sandbox changed, it will be killed and re-created.
71s (x14 over 4m4s)        Normal    SandboxChanged           Pod/cert-manager-startupapicheck-phzbc          Pod sandbox changed, it will be killed and re-created.
70s (x5371 over 3h15m)     Warning   FailedKillPod            Pod/cert-manager-866674c48b-brjt9               error killing pod: failed to "KillPodSandbox" for "5c5481a3-8dd0-430a-8397-9d1e087a1930" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"3e2f011819bc1e484ca31f255a6e6e97a5d8f5fc66679cc031e56b77659be888\": plugin type=\"multus\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: stat /etc/cni/net.d/multus.d/multus.kubeconfig: no such file or directory"
66s (x15 over 4m5s)        Normal    SandboxChanged           Pod/cert-manager-5f8646db6b-p8j2c               Pod sandbox changed, it will be killed and re-created.

[carrodher]

Please note Helm by design doesn't remove PVs/PVCs when uninstalling Helm charts, you need to manually clean up them.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.