[EKS] [EBS CSI addon]: Adding custom Toleration on EBS CSI Addon

[singhnix]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request Adding custom toleration like "node.tolerateAllTaints = true" on EBS CSI addon so that it can tolerate taints put on nodes.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

It seems like there is potentially a missing setting on the DaemonSet created by this Add-on that will prevent it from being put on certain nodes with taints on them. People who manage this manually have fixed this by adding:

node.tolerateAllTaints = true

to the daemonset. Currently it does not look like the AWS Add-on allows for something like this. Because of this our move over to using this add on has caused issues in new persistent volumes from being created on certain nodes.

I am not fully sure if there is a workaround for this (I assumed modifying the daemonset after install would not be permanent or should really be a step in setting up).

https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/848

Specifically the error we saw that led to this is:

Warning ProvisioningFailed 18s (x7 over 81s) ebs.csi.aws.com_ebs-csi-controller-848fb4bd69-lnjp8_454f66d4-704c-4164-a5a0-283cb99d5688 failed to provision volume with StorageClass "gp3-encrypted": error generating accessibility requirements: no topology key found on CSINode ip-XX-XXX-XX-XX.ec2.internal Normal ExternalProvisioning 8s (x6 over 81s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator

The label of:

topology.ebs.csi.aws.com/zone=<REGION>

was only on nodes that the daemonset could run on instead of every node (due to the daemonset).

Are you currently working around this issue? How are you currently solving this problem?

Additional context Anything else we should know?

Attachments Case ID:9858452761

[harrisjoseph]

Thank you for raising this ticket. I'm curious about why the current tolerations for the ebs-csi-node statefulset in the EBS CSI driver Add-on has been chosen?

We currently have this:

      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoExecute
        operator: Exists
        tolerationSeconds: 300

Which seems to assume that the user/operator won't want the ebs-csi-node, or any EBS functionality on any node that has any other taint set. Node isolation via taints is a pretty standard feature of k8s, it seems odd that we wouldn't want to use EBS volumes on all tainted nodes, when the daemonset could be extended to all nodes very easily:

      tolerations:
      - operator: Exists

I'm sure my question is just a sign that I have some deeper misunderstanding of how the EBS CSI driver is intended to be used in conjunction with EKS, I'm more than happy to be corrected, or to find that the CSI driver add-on can be easily configured to use EKS' standard topology keys, for instance.

[stephen-gardner]

I've been able to extend the daemonset to tainted nodes by modifying the tolerations, which is enough to deploy pending persistent volume claims...

kubectl -n kube-system get daemonset ebs-csi-node -o yaml --show-managed-fields | grep f:tolerations -C 2

            f:priorityClassName: {}
            f:serviceAccountName: {}
            f:tolerations: {}
            f:volumes:
              k:{"name":"device-dir"}:

Unfortunately, tolerations are a fully managed field, so my changes are quickly overwritten. The only workaround I could think of is to remove Amazon EBS CSI add-on, but preserve its resources, effectively turning it into a self-managed add-on.

eksctl delete addon --cluster my-cluster --name aws-ebs-csi-driver --preserve

[MartinEmrich]

Have the same issue. In the "standalone" ebs driver, this apparently can be configured with a helm variable "tolerateAllTaints", too: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/charts/aws-ebs-csi-driver/templates/node.yaml#L46

I have opened a ticket with AWS support on why they chose to set this to the (IMHO) less intuitive/sensible option.

[MartinEmrich]

Just to report back: AWS support recommended to use a self-managed installation of the CSI driver for the time being, and requested me to watch this issue on Github.

[ksquaredkey]

I'm in the same boat. An AddOn DaemonSet to provide a universal service that has by default an exclusion of any nodes with a Taint is like running Up the Down escalator. StatefulSets with PVs in a Production multi-AZ architecture - you're gonna have Taints on Worker Nodes so the SS Pods run in the right AZ. I see the option was added to the EBS CSI Driver 2 years ago. Haven't found any discussion around keeping the old default yet, so maybe I'm missing something important. But sure seems like an illogical choice given what EBS CSI Driver does.

[Chili-Man]

We just upgraded from an older helm chart version of the EBS CSI to the EKS managed add on version and did not realize that this problem until now. I don't understand and am disappointed that the add on does not allow the toleration field to be customizable for the daemonset - it makes absolutely no sense to not allow some configuration of it.

[idanl21]

Hey, facing the same issue. i want to add tolerations : - effect: NoSchedule operator:Exists ` to ebs-csi-node daemon set, but a few minutes after the change it deleted. i assuming that the ebs-csi-controller is causing it, but i cannot find the tolerations definition in the controller.

[MartinEmrich]

@idanl21 Indeed, currently the only solution is to remove the AWS-managed Add-In, and install the ebs-csi-controller yourself.

[GrigorievNick]

as I understand NodeSelector is also fullyManaged?

[nthienan]

this issue is preventing me to use AWS-managed addons. Any workarounds?

[dfquaresma]

I think I have solved this locally by editing ebs-csi-node daemonset by appending the following in the tolerations at line 365

- operator: Exists
  effect: NoSchedule

The toleration after the append will be like below:

      tolerations:
        - key: CriticalAddonsOnly
          operator: Exists
        - operator: Exists
          effect: NoExecute
          tolerationSeconds: 300
        - operator: Exists
          effect: NoSchedule

It's working, but I am almost sure that it will break again after the drive is updated.

[johnjeffers]

The solution above used drive on version 1.8. The latest version of the drive, 1.10, has already fixed the issue.

It's definitely not fixed in 1.10. I have 1.10 installed in my cluster and it won't deploy the daemonset to any nodes that have taints.

[dfquaresma]

It's definitely not fixed in 1.10. I have 1.10 installed in my cluster and it won't deploy the daemonset to any nodes that have taints.

@johnjeffers you are right, I checked only one of my clusters. The other one is still having the same issue. Thank you!

The workaround I shared before is still working.

[johnjeffers]

Could we please get an update on why this is taking so long to fix? All that has to be done is update the toleration on the daemonset to

      tolerations:
        - operator: Exists

The other EKS managed add-ons like kube-proxy and vpc-cni already do this. In fact, I copied and pasted that code block directly from the kube-proxy managed add-on's daemonset.

This is a blocker for upgrading to EKS 1.23, and the fix seems to be so simple. I can't understand why this issue has been open for so long. Is there more to this problem than there seems to be?

[sriramranganathan]

We recently released a behavior change that will NOT overwrite configuration changes made to EKS managed add-ons through the Kubernetes API. Previously, a reconciliation process ran every 15 minutes that overwrote configuration changes made to EKS managed add-ons through the Kubernetes API. Example – changes you make to the CoreDNS Config Map through the Kubernetes API will no longer be overwritten during steady state. However, if a managed add-on is upgraded, then any configuration changes made will not be retained at this time.

This change is a first step in ensuring configurations made to EKS add-ons are preserved. We are also working on additional changes to support advanced configuration of EKS add-ons directly through the EKS API, and the ability to preserve the configuration changes during add-on upgrades.

Toleration for the EBS-CSI driver is in our product backlog and is being evaluated by the team.

[johnjeffers]

OK just to make sure I understand, the permanent fix for this is in the backlog so we won't see it for a while, but if I manually update the daemonset with the tolerations I need, nothing will overwrite my changes until the next upgrade of the add-on?

[youwalther65]

OK just to make sure I understand, the permanent fix for this is in the backlog so we won't see it for a while, but if I manually update the daemonset with the tolerations I need, nothing will overwrite my changes until the next upgrade of the add-on?

Yes, I already tested it applying the requested toleration and it wasn't reconciled. Upgrading will lead to a merge conflict though and you have to use OVERWRITE and re- apply your previous change(s).

[Joseph-Irving]

As more and more people move to K8s 1.23 where the ebs-csi-driver is mandatory, I imagine a fair amount will try out this addon and realise it doesn't work nicely with taints. It's a shame as we would like to use the eks-addons where possible and it seems like AWS is recommending people do that. Other Daemonsets that are eks-addons don't have this issue, so I'm surprised this one acts differently, I would've thought most people would consider EBS mounting something that they would want on all nodes by default as that's how it works pre 1.23.

[ksliu58]

Thanks for the feedback everyone.

Currently, you can modify your tolerations without been overwritten by EKS. We will be updating the CSI driver with EKS managed add-on in the next release, updating the daemonset to allow custom tolerations for all taints by default.

[cilindrox]

Just got bit by the same issue. Not having at least a warning on the docs or troubleshooting docs made things unnecessarily complicated. Having tolerations for every taint should be the default behavior for the DS.

[youwalther65]

Amazon has just released EKS managed add-on EBS CSI "v1.11.2-eksbuild.1". The new default for tolerations of DaemonSet "ebs-csi-node" was changed according to the requests from customers and community to:

$ k get ds -n kube-system ebs-csi-node -o yaml
...
      tolerations:
      - operator: Exists
...

This is now similar to other add-ons like kube-proxy or AWS VPC CNI.

[GNSunny]

I don't see it in the AWS console, nor from the AWS CLI.

[youwalther65]

I don't see it in the AWS console, nor from the AWS CLI.

I tested in in eu-west-1 (Dublin/Ieland) in the morning. Usually it is a phased rollout per region. You should be able to see it in a couple of hours. Please re-try later!

[GNSunny]

I have tried at 16:00 BST from eu-west-1 region too. I didn't see any updated version, but I'll retry later

[aleclerc-sonrai]

Does the cluster need to be at 1.23 to get v1.11.2-eksbuild.1?

[Chili-Man]

@aleclerc-sonrai I'm on 1.23 and I don't see the new version yet either (us-west-2)

[ksquaredkey]

Still not available in us-west-2. Latest from "aws eks describe-addon-versions" is still: "v1.10.0-eksbuild.1". If I hadn't been so busy with other higher priority tickets, I'd have punted on waiting for the add-on and install and manage the drivers myself. The add-on is worthless with the Taint issue.

[ksliu58]

Thanks everyone for your patience. The upcoming release with the new default for tolerations of DaemonSet is in progress, now with 15 out of 27 regions completed. We will complete deployment in the remaining regions shortly.

[duclm2609]

I hit the same problem and is waiting for new version rollout to finish to continue my work. Current ap-southeast-1 region is not available yet.

[parraletz]

@duclm2609 upgrade eks addon to v1.11.2-eksbuild.1 , in this version has been resolved the issue

My env:

eks cluster : kubernetes 1.23 version region: oregon