IRSA setup for EKS managed add-on "vpc-cni" without relying on node IAM permission does not always work [Bug]

[youwalther65]

What were you trying to accomplish?

Create an EKS cluster with IRSA setup for EKS managed add-on "vpc-cni" without relying on node IAM permission: AmazonEKS_CNI_Policy

What happened?

Sometimes ServiceAccount "aws-node" which is used by corresponding DaemonSet "aws-node" is not IRSA annotated and DaemonSet aws-node is not fully working because of missing IAM permissions (and node does not have the IAM permission because it is excluded deliberately for least privilege). But sometimes it works and SA is annotated but it's not reliable. I want to choose least privilege and avoid adding the CNI permissions to node profile.

How to reproduce it?

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: simple-cluster
  version: "1.22"
  region: eu-west-1

kubernetesNetworkConfig:
  ipFamily: IPv4

vpc:
  clusterEndpoints:
    publicAccess: true
    privateAccess: false

iam:
  withOIDC: true

addons:
  - name: vpc-cni
#    use default version, should be v1.10.1-eksbuild.1 currently
#    version: v1.11.2-eksbuild.1
    attachPolicyARNs:
    - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
  - name: aws-ebs-csi-driver
    version: v1.10.0-eksbuild.1
    attachPolicyARNs:
    - arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy

managedNodeGroups:
  - name: mng-spot
    amiFamily: AmazonLinux2
    instanceSelector:
      vCPUs: 2
      memory: "4" # 4 GiB, unit defaults to GiB
      cpuArchitecture: x86_64 # default value
    minSize: 1
    desiredCapacity: 2
    maxSize: 2
    volumeSize: 50
    volumeType: gp3
    volumeEncrypted: true
    privateNetworking: true
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        # without this policy you might run into deadlocks - see opened issue ???
#        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    spot: true
    labels:
      my-cool-label: pizza
      owner: waltju
#    don't use taints because EKS managed add-on EBS CSI will not work with that
#    taints:
#      - key: feaster
#        value: "true"
#        effect: NoSchedule
    tags:
      k8s.io/cluster-autoscaler/node-template/label/owner: "waltju"
      k8s.io/cluster-autoscaler/node-template/taint/feaster: "true:NoSchedule"
      auto-delete: "no"
    propagateASGTags: true

Logs

$ k get sa -n kube-system aws-node -o yaml | head -7
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"aws-node","namespace":"kube-system"}}
  creationTimestamp: "2022-08-11T05:40:25Z"
...

Anything else we need to know?

Versions

$ eksctl info

$ eksctl info eksctl version: 0.107.0 kubectl version: v1.23.5 OS: linux

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.