[EKS] [request]: Add default toleration for Pod identity agent add-on

[ssup2]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request Please add default toleration for Pod identity agent add-on

tolerations: 
- operator: Exists

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Pod identity agent add-on aren't deploying on all nodes due to node taints and lack of default tolerations.

Are you currently working around this issue? None

Additional context None

Attachments Similar Issue : https://github.com/aws/containers-roadmap/issues/2195

[mikestef9]

tolerations is an available field in the configuration schema for the add-on. Here is an example defining the add-on with CloudFormation that sets a toleration to tolerate anything

  EKSPodIdentityAgent:
    Type: AWS::EKS::Addon
    Properties:
      AddonName: "eks-pod-identity-agent"
      AddonVersion: "v1.1.0-eksbuild.1"
      ClusterName: my-cluster
      ConfigurationValues: |-
        resources:
          limits:
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
        tolerations:
          - operator: Exists
      ResolveConflicts: "OVERWRITE"

[mikestef9]

With that said, this should probably become the default setting, so leaving this issue open to track that.

[ssup2]

Add-ons distributed through DaemonSet seem to need unification of default values. This is because default toleration is set in the case of VPC CNI. From a customer perspective, this can be confusing. My customer naturally expected that it would be distributed to all nodes regardless of the node's taint.

[bitshop]

As a customer I definitely expected this to work, but also understand @mikestef9's response - however often, and in my case today, the EKS blueprints were done via CDK by the infrastructure group - we control the karpenter NodePool and EC2NodeClass and therefore, as configured by default, can't enable this to work and get blocked when we thought we were simply changing some taints/tolerations in our config.

[georgejohnis]

The versions v1.2.0-eksbuild.1 and above of Amazon EKS Pod Identity Agent Add-on has been updated with default toleration. Hence closing this issue. Feel free to reopen if you encounter any issues.