EKS with Fargate Profile CoreDNS annotation modifications

[chandankashyap19]

Hello,

EKS with Fargate profile is one of the next levels of K8 enhancing and make it "serverless". But, due to lack of a few minor modifications, it causes a lot of issues.

We are using Terraform as IAC to provision our EKS with fargate profile cluster but CoreDNS is not getting provisioned because it looks for Node group to spin up the pods. Interesting point is the "annotation" of coreDNS which is still pointing to "compute-type". Only way to run codeDNS on profile is "patching". It is one of a workaround. Not a good solution from IAC point of view.

AWS has given its ekctl command to provision cluster and --fargate in command do the patching work but if you are using terraform then there is no any provider given to do that. Terraform community pointing to the AWS team because there is no any API given by AWS to achieve the same.

I would suggest AWS team to provide a way to do same patching of coreDNS in behind (like ekctl does) even by using terraform. It would help many people.

#EKS with Fargate

[mogul]

Just ran into this myself and resolved it with a workaround over here. The key bit is waiting for the Fargate profile using depends_on before restarting the patched coredns pod.

[silverlyra]

I noticed this is still true in EKS 1.21. I am confused why this annotation is applied to the CoreDNS pods in the first place – am I right in that CoreDNS would not be launched on Fargate without a matching Fargate profile first being applied to the cluster? Could AWS just remove this annotation from the CoreDNS pods without ill effect?

(I’m using the Terraform workaround suggested above, and it works but it’s quite a kludge.)

[fmmasood]

Is there any update on this please? (if possible and backward compatible) Removing this annotation will make it way simpler to run coredns on EKS Fargate. This inturn will make it easier to deploy other components like karpenter, ADOT directly to fargate only cluster.

[mrvisser]

AWS has some guidance on how to automate this workaround in a platform-agnostic way using Lambda:

https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/deploy-coredns-on-amazon-eks-with-fargate-automatically-using-terraform-and-python.html

Which is... something.

I've had success for similar one-offs using terraform by using a K8s Job resource, but I'm not sure if a kubectl container could be rigged to invoke cluster commands without having coredns available yet. Has anyone tried, or is anyone willing to try it as an improvement over the local-exec provisioner approach?

[bryantbiggs]

This is now available - here is an example that demonstrates https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/fargate_profile/main.tf#L36-L40

[sriramranganathan]

Amazon EKS team recently announced the general availability of advanced configuration feature for managed add-ons. You can now pass in advanced configuration for cluster add-ons, enabling you to customize add-on properties not handled by default settings. Configuration can be applied to add-ons either during cluster creation or at any time after the cluster is created.

Using advanced configuration feature, you can now specify the computeType as Fargate for the CoreDNS add-on.

To learn more about this feature, check out this blogpost - https://aws.amazon.com/blogs/containers/amazon-eks-add-ons-advanced-configuration/

Check out the Amazon EKS documentation - https://docs.aws.amazon.com/eks/latest/userguide/managing-add-ons.html