multus-cni
multus-cni copied to clipboard
multus kubeconfig file not populated correctly after node reboot
What happend: Apiserver IP address was empty in multus kubeconfig file
/etc/cni/net.d/multus.d/multus.kubeconfig (server: https://[])
`apiVersion: v1
kind: Config
clusters:
- name: local
cluster:
server: https://[]:
certificate-authority-data: [snipped]==
users:
- name: multus
user:
token: "[snipped]=="
contexts:
- name: multus-context
context:
cluster: local
user: multus
current-context: multus-context`
Pods What you expected to happen: Pods were not scheduled and the below errors were appearing in /var/log/messages
May 20 14:13:20 ace-func1-3n1 kubelet[1683]: E0520 14:13:20.399208 1683 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"d4fd3ede-c09e-4a3f-abd2-8eeb6ba0b3da\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"d02259dca386dfd70b0d6433452e99a880b57e849f79f74993ace14d90e55fb9\\\": plugin type=\\\"multus\\\" name=\\\"multus-cni-network\\\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: error loading config file \\\"/etc/cni/net.d/multus.d/multus.kubeconfig\\\": yaml: line 7: mapping values are not allowed in this context\"" pod="kube-system/coredns-58f4964b57-f8wkh" podUID="d4fd3ede-c09e-4a3f-abd2-8eeb6ba0b3da"
How to reproduce it (as minimally and precisely as possible): Reboot a node in multi-node Kubernetes cluster
Anything else we need to know?:
Environment:
- Multus version : ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
- Kubernetes version (use
kubectl version
):Client Version: v1.29.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.1
- Primary CNI for Kubernetes cluster: Calico
- OS (e.g. from /etc/os-release): ` NAME="Red Hat Enterprise Linux" VERSION="9.3 (Plow)" ID="rhel" ID_LIKE="fedora" VERSION_ID="9.3" PLATFORM_ID="platform:el9" PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)" ANSI_COLOR="0;31" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9" BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9" REDHAT_BUGZILLA_PRODUCT_VERSION=9.3 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.3" `
- File of '/etc/cni/net.d/'
- File of '/etc/cni/multus/net.d'
- NetworkAttachment info (use
kubectl get net-attach-def -o yaml
) - Target pod yaml info (with annotation, use
kubectl get pod <podname> -o yaml
) ` apiVersion: v1 kind: Pod metadata: creationTimestamp: "2024-05-20T09:02:12Z" generateName: kube-multus-ds- labels: app: multus controller-revision-hash: 789c4467b8 name: multus pod-template-generation: "1" tier: node name: kube-multus-ds-hxtkt namespace: kube-system ownerReferences:- apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: DaemonSet name: kube-multus-ds uid: cc6cd4f8-540a-4593-b099-4be99d0088bd resourceVersion: "2107922" uid: 308c615b-8284-4c79-91f5-9c3ec976e0e6 spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchFields: - key: metadata.name operator: In values: - ace-func1-3n1 containers:
- args:
- --cni-version=0.3.1
- --cni-conf-dir=/host/etc/cni/net.d
- --multus-autoconfig-dir=/host/etc/cni/net.d
- --multus-log-to-stderr=true
- --multus-log-level=verbose command:
- /thin_entrypoint image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2 imagePullPolicy: IfNotPresent name: kube-multus resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
- mountPath: /host/etc/cni/net.d name: cni
- mountPath: /host/opt/cni/bin name: cnibin
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-5wxv9 readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true hostNetwork: true initContainers:
- args:
- --type
- thin command:
- /install_multus image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2 imagePullPolicy: IfNotPresent name: install-multus-binary resources: requests: cpu: 10m memory: 15Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
- mountPath: /host/opt/cni/bin mountPropagation: Bidirectional name: cnibin
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-5wxv9 readOnly: true nodeName: ace-func1-3n1 preemptionPolicy: PreemptLowerPriority priority: 2000001000 priorityClassName: system-node-critical restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: multus serviceAccountName: multus terminationGracePeriodSeconds: 10 tolerations:
- effect: NoSchedule operator: Exists
- effect: NoExecute operator: Exists
- effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists
- effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists
- effect: NoSchedule key: node.kubernetes.io/disk-pressure operator: Exists
- effect: NoSchedule key: node.kubernetes.io/memory-pressure operator: Exists
- effect: NoSchedule key: node.kubernetes.io/pid-pressure operator: Exists
- effect: NoSchedule key: node.kubernetes.io/unschedulable operator: Exists
- effect: NoSchedule key: node.kubernetes.io/network-unavailable operator: Exists volumes:
- hostPath: path: /etc/cni/net.d type: "" name: cni
- hostPath: path: /opt/cni/bin type: "" name: cnibin
- name: kube-api-access-5wxv9
projected:
defaultMode: 420
sources:
- serviceAccountToken: expirationSeconds: 3607 path: token
- configMap:
items:
- key: ca.crt path: ca.crt name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions:
- lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:13Z" status: "True" type: PodReadyToStartContainers
- lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:14Z" status: "True" type: Initialized
- lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:15Z" status: "True" type: Ready
- lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:15Z" status: "True" type: ContainersReady
- lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:12Z" status: "True" type: PodScheduled containerStatuses:
- containerID: containerd://fe1413443a789d09e25900c72cd3cfc2d33a6ccd5308bf75741284952ed58d18 image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2 imageID: sha256:2ce3492d2e6f598566f6ec26d08c39ca8c43df5beeefbb7ce38a1a4ef3809760 lastState: {} name: kube-multus ready: true restartCount: 0 started: true state: running: startedAt: "2024-05-20T09:02:14Z" hostIP: 10.52.214.11 hostIPs:
- ip: 10.52.214.11 initContainerStatuses:
- containerID: containerd://9e4195df464d4d98784b1d7521d5d71a775a34f334e0dc889e7f366171f8a0e3 image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2 imageID: sha256:2ce3492d2e6f598566f6ec26d08c39ca8c43df5beeefbb7ce38a1a4ef3809760 lastState: {} name: install-multus-binary ready: true restartCount: 0 started: false state: terminated: containerID: containerd://9e4195df464d4d98784b1d7521d5d71a775a34f334e0dc889e7f366171f8a0e3 exitCode: 0 finishedAt: "2024-05-20T09:02:13Z" reason: Completed startedAt: "2024-05-20T09:02:13Z" phase: Running podIP: 10.52.214.11 podIPs:
- ip: 10.52.214.11 qosClass: Burstable startTime: "2024-05-20T09:02:12Z" `
- Other log outputs (if you use multus logging)