multus-cni icon indicating copy to clipboard operation
multus-cni copied to clipboard

Published 20 hours ago • verified publisher icon k8snetworkplumbingwg

multus kubeconfig file not populated correctly after node reboot

Open geeky-akshay opened this issue 3 months ago • 1 comments

What happend: Apiserver IP address was empty in multus kubeconfig file

/etc/cni/net.d/multus.d/multus.kubeconfig (server: https://[])
`apiVersion: v1
kind: Config
clusters:
- name: local
  cluster:
    server: https://[]:
    certificate-authority-data: [snipped]==
users:
- name: multus
  user:
    token: "[snipped]=="
contexts:
- name: multus-context
  context:
    cluster: local
    user: multus
current-context: multus-context`

Pods What you expected to happen: Pods were not scheduled and the below errors were appearing in /var/log/messages

May 20 14:13:20 ace-func1-3n1 kubelet[1683]: E0520 14:13:20.399208 1683 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"d4fd3ede-c09e-4a3f-abd2-8eeb6ba0b3da\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"d02259dca386dfd70b0d6433452e99a880b57e849f79f74993ace14d90e55fb9\\\": plugin type=\\\"multus\\\" name=\\\"multus-cni-network\\\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: error loading config file \\\"/etc/cni/net.d/multus.d/multus.kubeconfig\\\": yaml: line 7: mapping values are not allowed in this context\"" pod="kube-system/coredns-58f4964b57-f8wkh" podUID="d4fd3ede-c09e-4a3f-abd2-8eeb6ba0b3da"

How to reproduce it (as minimally and precisely as possible): Reboot a node in multi-node Kubernetes cluster

Anything else we need to know?:

Environment:

  • Multus version : ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
  • Kubernetes version (use kubectl version): Client Version: v1.29.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.1
  • Primary CNI for Kubernetes cluster: Calico
  • OS (e.g. from /etc/os-release): ` NAME="Red Hat Enterprise Linux" VERSION="9.3 (Plow)" ID="rhel" ID_LIKE="fedora" VERSION_ID="9.3" PLATFORM_ID="platform:el9" PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)" ANSI_COLOR="0;31" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9" BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9" REDHAT_BUGZILLA_PRODUCT_VERSION=9.3 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="9.3" `

  • File of '/etc/cni/net.d/'
  • File of '/etc/cni/multus/net.d'
  • NetworkAttachment info (use kubectl get net-attach-def -o yaml)
  • Target pod yaml info (with annotation, use kubectl get pod <podname> -o yaml) ` apiVersion: v1 kind: Pod metadata: creationTimestamp: "2024-05-20T09:02:12Z" generateName: kube-multus-ds- labels: app: multus controller-revision-hash: 789c4467b8 name: multus pod-template-generation: "1" tier: node name: kube-multus-ds-hxtkt namespace: kube-system ownerReferences:
    • apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: DaemonSet name: kube-multus-ds uid: cc6cd4f8-540a-4593-b099-4be99d0088bd resourceVersion: "2107922" uid: 308c615b-8284-4c79-91f5-9c3ec976e0e6 spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchFields: - key: metadata.name operator: In values: - ace-func1-3n1 containers:
    • args:
      • --cni-version=0.3.1
      • --cni-conf-dir=/host/etc/cni/net.d
      • --multus-autoconfig-dir=/host/etc/cni/net.d
      • --multus-log-to-stderr=true
      • --multus-log-level=verbose command:
      • /thin_entrypoint image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2 imagePullPolicy: IfNotPresent name: kube-multus resources: limits: cpu: 100m memory: 50Mi requests: cpu: 100m memory: 50Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
      • mountPath: /host/etc/cni/net.d name: cni
      • mountPath: /host/opt/cni/bin name: cnibin
      • mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-5wxv9 readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true hostNetwork: true initContainers:
    • args:
      • --type
      • thin command:
      • /install_multus image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2 imagePullPolicy: IfNotPresent name: install-multus-binary resources: requests: cpu: 10m memory: 15Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts:
      • mountPath: /host/opt/cni/bin mountPropagation: Bidirectional name: cnibin
      • mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-5wxv9 readOnly: true nodeName: ace-func1-3n1 preemptionPolicy: PreemptLowerPriority priority: 2000001000 priorityClassName: system-node-critical restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: multus serviceAccountName: multus terminationGracePeriodSeconds: 10 tolerations:
    • effect: NoSchedule operator: Exists
    • effect: NoExecute operator: Exists
    • effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists
    • effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists
    • effect: NoSchedule key: node.kubernetes.io/disk-pressure operator: Exists
    • effect: NoSchedule key: node.kubernetes.io/memory-pressure operator: Exists
    • effect: NoSchedule key: node.kubernetes.io/pid-pressure operator: Exists
    • effect: NoSchedule key: node.kubernetes.io/unschedulable operator: Exists
    • effect: NoSchedule key: node.kubernetes.io/network-unavailable operator: Exists volumes:
    • hostPath: path: /etc/cni/net.d type: "" name: cni
    • hostPath: path: /opt/cni/bin type: "" name: cnibin
    • name: kube-api-access-5wxv9 projected: defaultMode: 420 sources:
      • serviceAccountToken: expirationSeconds: 3607 path: token
      • configMap: items:
        • key: ca.crt path: ca.crt name: kube-root-ca.crt
      • downwardAPI: items:
        • fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions:
    • lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:13Z" status: "True" type: PodReadyToStartContainers
    • lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:14Z" status: "True" type: Initialized
    • lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:15Z" status: "True" type: Ready
    • lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:15Z" status: "True" type: ContainersReady
    • lastProbeTime: null lastTransitionTime: "2024-05-20T09:02:12Z" status: "True" type: PodScheduled containerStatuses:
    • containerID: containerd://fe1413443a789d09e25900c72cd3cfc2d33a6ccd5308bf75741284952ed58d18 image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2 imageID: sha256:2ce3492d2e6f598566f6ec26d08c39ca8c43df5beeefbb7ce38a1a4ef3809760 lastState: {} name: kube-multus ready: true restartCount: 0 started: true state: running: startedAt: "2024-05-20T09:02:14Z" hostIP: 10.52.214.11 hostIPs:
    • ip: 10.52.214.11 initContainerStatuses:
    • containerID: containerd://9e4195df464d4d98784b1d7521d5d71a775a34f334e0dc889e7f366171f8a0e3 image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2 imageID: sha256:2ce3492d2e6f598566f6ec26d08c39ca8c43df5beeefbb7ce38a1a4ef3809760 lastState: {} name: install-multus-binary ready: true restartCount: 0 started: false state: terminated: containerID: containerd://9e4195df464d4d98784b1d7521d5d71a775a34f334e0dc889e7f366171f8a0e3 exitCode: 0 finishedAt: "2024-05-20T09:02:13Z" reason: Completed startedAt: "2024-05-20T09:02:13Z" phase: Running podIP: 10.52.214.11 podIPs:
    • ip: 10.52.214.11 qosClass: Burstable startTime: "2024-05-20T09:02:12Z" `
  • Other log outputs (if you use multus logging)

geeky-akshay avatar May 20 '24 10:05 geeky-akshay