Cannot start calico-controller on AWS EKS

[lkoniecz]

CNI-genie installed from following spec: kubectl apply -f https://raw.githubusercontent.com/cni-genie/CNI-Genie/master/conf/1.8/genie-plugin.yaml

I instlled calico afterwards:https://docs.projectcalico.org/v3.7/getting-started/kubernetes/installation/calico#installing-with-the-kubernetes-api-datastore50-nodes-or-less using the < 50 ndoes spec.

calico-controller wont start up:

kubectl describe pod calico-kube-controllers-f59b77565-gj8k5
Name:               calico-kube-controllers-f59b77565-gj8k5
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-51-181-180.ec2.internal/10.51.181.180
Start Time:         Tue, 15 Oct 2019 09:23:55 +0200
Labels:             k8s-app=calico-kube-controllers
                    pod-template-hash=f59b77565
Annotations:        kubernetes.io/psp: eks.privileged
Status:             Pending
IP:                 
Controlled By:      ReplicaSet/calico-kube-controllers-f59b77565
Containers:
  calico-kube-controllers:
    Container ID:   
    Image:          calico/kube-controllers:v3.7.5
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Readiness:      exec [/usr/bin/check-status -r] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      ENABLED_CONTROLLERS:  node
      DATASTORE_TYPE:       kubernetes
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from calico-kube-controllers-token-cmrkj (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  calico-kube-controllers-token-cmrkj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  calico-kube-controllers-token-cmrkj
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                From                                    Message
  ----     ------                  ----               ----                                    -------
  Normal   Scheduled               54s                default-scheduler                       Successfully assigned kube-system/calico-kube-controllers-f59b77565-gj8k5 to ip-10-51-181-180.ec2.internal
  Warning  FailedCreatePodSandBox  53s                kubelet, ip-10-51-181-180.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "7fb1f1c7d7bd068450b2698caaf0ff6cfd52745f3c91e8a2c82b33f4e98ffa16" network for pod "calico-kube-controllers-f59b77565-gj8k5": NetworkPlugin cni failed to set up pod "calico-kube-controllers-f59b77565-gj8k5_kube-system" network: CNI Genie Add IP internal error: CNI Genie error at ParsePodAnnotations: Corresponding binary for user requested plugin (weave) is not present in plugin directory (/opt/cni/bin), result: %!!(MISSING)s(<nil>), failed to clean up sandbox container "7fb1f1c7d7bd068450b2698caaf0ff6cfd52745f3c91e8a2c82b33f4e98ffa16" network for pod "calico-kube-controllers-f59b77565-gj8k5": NetworkPlugin cni failed to teardown pod "calico-kube-controllers-f59b77565-gj8k5_kube-system" network: CNI Genie release IP internal error: CNI Genie error at ParsePodAnnotations: Corresponding binary for user requested plugin (weave) is not present in plugin directory (/opt/cni/bin)]
  Normal   SandboxChanged          13s (x5 over 52s)  kubelet, ip-10-51-181-180.ec2.internal  Pod sandbox changed, it will be killed and re-created.

Looks like cni-genie seeks for weave-net binary, but the node does not have one:

[ec2-user@ip-10-51-181-180 net.d]$ ls /opt/cni/bin
aws-cni  aws-cni-support.sh  bridge  calico  calico-ipam  cnitool  dhcp  flannel  genie  host-device  host-local  ipvlan  loopback  macvlan  noop  portmap  ptp  sample  tuning  vlan

kubelet status:

[ec2-user@ip-10-51-181-180 net.d]$ systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubelet-args.conf
   Active: active (running) since pią 2019-10-11 11:37:34 UTC; 3 days ago
     Docs: https://github.com/kubernetes/kubernetes
  Process: 3763 ExecStartPre=/sbin/iptables -P FORWARD ACCEPT (code=exited, status=0/SUCCESS)
 Main PID: 3774 (kubelet)
    Tasks: 26
   Memory: 162.7M
   CGroup: /system.slice/kubelet.service
           └─3774 /usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/kubelet-config.json --allow-privileged=true --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime docker --network-plugin cni --node-ip=10.51.181.180 --pod-infra-container-image=60240...

paź 15 07:27:32 ip-10-51-181-180.ec2.internal kubelet[3774]: CNI Genie workloadID= kube-system.calico-kube-controllers-f59b77565-gj8k5
paź 15 07:27:32 ip-10-51-181-180.ec2.internal kubelet[3774]: CNI Genie orchestratorID= k8s
paź 15 07:27:32 ip-10-51-181-180.ec2.internal kubelet[3774]: CNI Genie annot= [map[kubernetes.io/psp:eks.privileged]]
paź 15 07:27:32 ip-10-51-181-180.ec2.internal kubelet[3774]: CNI Genie no annotations is given! Using default plugins: [weave],  annot is map[kubernetes.io/psp:eks.privileged]
paź 15 07:27:32 ip-10-51-181-180.ec2.internal kubelet[3774]: CNI Genie plugion map: map[weave:map[false:[1]]]
paź 15 07:27:32 ip-10-51-181-180.ec2.internal kubelet[3774]: E1015 07:27:32.960889    3774 cni.go:345] Error deleting kube-system_calico-kube-controllers-f59b77565-gj8k5/7fb1f1c7d7bd068450b2698caaf0ff6cfd52745f3c91e8a2c82b33f4e98ffa16 from network genie/k8s-pod-network: CNI Gen...
paź 15 07:27:32 ip-10-51-181-180.ec2.internal kubelet[3774]: E1015 07:27:32.961490    3774 remote_runtime.go:119] StopPodSandbox "7fb1f1c7d7bd068450b2698caaf0ff6cfd52745f3c91e8a2c82b33f4e98ffa16" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni fa...
paź 15 07:27:32 ip-10-51-181-180.ec2.internal kubelet[3774]: E1015 07:27:32.961695    3774 kuberuntime_manager.go:815] Failed to stop sandbox {"docker" "7fb1f1c7d7bd068450b2698caaf0ff6cfd52745f3c91e8a2c82b33f4e98ffa16"}
paź 15 07:27:32 ip-10-51-181-180.ec2.internal kubelet[3774]: E1015 07:27:32.961910    3774 kuberuntime_manager.go:610] killPodWithSyncResult failed: failed to "KillPodSandbox" for "bf0e9036-ef1c-11e9-8afa-120b23f0336a" with KillPodSandboxError: "rpc error: co...ontrollers-f59b7756
paź 15 07:27:32 ip-10-51-181-180.ec2.internal kubelet[3774]: E1015 07:27:32.962947    3774 pod_workers.go:190] Error syncing pod bf0e9036-ef1c-11e9-8afa-120b23f0336a ("calico-kube-controllers-f59b77565-gj8k5_kube-system(bf0e9036-ef1c-11e9-8afa-120b23f0336a)")...336a" with KillPodS
Hint: Some lines were ellipsized, use -l to show in full.

[ianhe8x]

you need to add annotation on calico-kube-controllers's deployment

annotations: 
  cni: "calico"