[EKS] [request]: Support CoreDNS configuration via additional configmap

[andrewpurdin]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request We would like to be able to customize the CoreDNS configuration without having to hand edit the default configmap that is created for it. Although editing the configmap is a valid approach in some cases, it is an anti-pattern for infrastructure-as-code deployments since the resource is not managed in IaC.

AKS has solved this by exposing an additional configmap that one can optionally create and manage which will be imported as CoreDNS configuration. This allows users to create that additional configmap and manage it under IaC post cluster creation.

#1159 may be a prerequisite for adding this functionality, but I am not sure.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Ultimately. we simply want to manage CoreDNS configuration with our IaC tooling. Until we can do that, we either have to avoid updating the CoreDNS configuration, or come up with specialized configurations to handle this case outside of IaC.

Are you currently working around this issue? We currently don't have a good workaround for this issue.

[bengaywins]

@andrewpurdin Why can't you just use Kustomize to do this exact thing? It would be IaC in declarative form that allows for you to run kubectl apply -k /folder/to/your/kustomize/overlays

[andrewpurdin]

@andrewpurdin Why can't you just use Kustomize to do this exact thing? It would be IaC in declarative form that allows for you to run kubectl apply -k /folder/to/your/kustomize/overlays

I've not used Kustomize personally, perhaps we could adopt the resource after creation time with something like this. However it would still be much simpler if we could manage an object at creation time rather than having to take extra steps afterward.

[bengaywins]

The thing is, at creation time doesn't really exist in the sense of before it creates. So it's always a step after anyway. Unless they were to avoid deploying CoreDNS entirely. Which we both know is a disaster waiting to happen.

Kustomize is rapidly becoming the suggested way to deploy all things into a K8s cluster. It's a lot better than Helm and doesn't require special knowledge.

If you don't mind saying, what IaC paradigm are you using? If Terraform, there is a Kustomize provider you can use. I can't say I have used it yet, but likely will soon.

[andrewpurdin]

We're using pulumi, and I hear what you're saying about creation time, however we are able to manage the auth configmap which is created during the cluster creation process so I don't see why a configmap for controlling CoreDNS couldn't be exposed in a similar fashion. @mikestef9 can you confirm if this is a feature AWS would like to support?

[bengaywins]

https://www.pulumi.com/blog/announcing-kustomize-support/

[stevehipwell]

@bengaywins I'm not sure of the relevance of Kustomize in this as all it's doing is patching an existing object but without the fine grained control of an actual IaC tool (using Kustomize from Terraform or Pulumi might be useful as part of a migration but doesn't make a lot of sense as an actual pattern). This could be achieved by multiple means but all solutions are still just patching an EKS managed object with an unknown reconciliation loop.

@andrewpurdin here is issue https://github.com/aws/containers-roadmap/issues/923 to create a bare cluster allowing full control of everything installed.

[matti]

related #1679

[oguzhanaygn]

I'm running EKS 1.21 with managed add-ons. When my pods are starts to downscale, some No Route to Host errors are coming from the application pods. I'm suspecting the CoreDNS cache, but I couldn't examine it because CoreDNS doesn't allow its configmap to be reconfigured. So I'm kinda stuck because of this situation.

[Istador]

One possible workaround for now is to migrate from managed to self-managed coredns. Then you can edit the config as you please, without it getting overwritten automatically.

To switch to self-managed, you can delete the coredns add-on with the --preserve flag. This will keep all coredns resources within the cluster as they are at that moment. Afterwards you can modify the configmap. (If I remember correctly, there even is a checkbox for --preserve when deleting via the console.) See https://github.com/aws/containers-roadmap/issues/1415#issuecomment-904770014

Once the feature of this issue is implemented, it's very simple to just install the add-on again (make sure to backup your custom config somewhere or it might get overwritten then).

[skupjoe]

One possible workaround for now is to migrate from managed to self-managed coredns. Then you can edit the config as you please, without it getting overwritten automatically.

To switch to self-managed, you can delete the coredns add-on with the --preserve flag. This will keep all coredns resources within the cluster as they are at that moment. Afterwards you can modify the configmap. (If I remember correctly, there even is a checkbox for --preserve when deleting via the console.) See #1415 (comment)

Once the feature of this issue is implemented, it's very simple to just install the add-on again (make sure to backup your custom config somewhere or it might get overwritten then).

Right- this is the only way currently to be able to update EKS add-on configmaps or deployment/daemonset manifests, due to the presence of the eks manager which periodically updates managedFields, thereby overriding any custom changes that you've made.

It's unfortunate that you must deregister from AWS to do this, and I'm hoping in the future there will be a better way.

[maximmold]

Since coredns is added by default but doesn't show up in the list of addons, do you have to add it and then remove it with the reserve flag?

[sriramranganathan]

Amazon EKS team recently announced the general availability of advanced configuration feature for managed add-ons. You can now pass in advanced configuration for cluster add-ons, enabling you to customize add-on properties not handled by default settings. Configuration can be applied to add-ons either during cluster creation or at any time after the cluster is created.

Custom configuration will also allow you to customize the CoreDNS addon by allowing modification of the corefile.

To learn more about this feature, check out this blogpost - https://aws.amazon.com/blogs/containers/amazon-eks-add-ons-advanced-configuration/

Check out the Amazon EKS documentation - https://docs.aws.amazon.com/eks/latest/userguide/managing-add-ons.html

[pcebul]

Hey @sriramranganathan @mikestef9

Sorry to bother you, but I cant find a single example on the internet... using aws eks describe-addon-configuration I can see under configurationSchema that I should be able to deploy CoreDNS addon with my custom nodeSelector, and I want to do this using CloudFormation template. But like I mentioned, I cant find a single example on web how to use ConfigurationValues field (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-eks-addon.html#cfn-eks-addon-configurationvalues) . Also, how can I check what are the defaults for managed fields (before deploying it), eg what are the defaults for tolerations per specific addon version? Thanks!

[sriramranganathan]

Hi @pcebul . The blogpost should help you get started with using configurationValues when creating or updating an addon. While the blog does not show the implementation from AWS CloudFormation, the approach is similar wherein you can pass in configurationValues as a String value. Today, the configuration schema for CoreDNS addon does not support specifying tolerations but this is something we can enhance and as part of the schema, show the default values as well. The JSON schema has provision to show the default values for different parameters.