[FARGATE] [request]: DNS configuration for containers

[nhlushak]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request As for now, you can not override container dns settings for ecs tasks using awsvpc network mode, so you stuck with default VPC resolver or DHCP options set.

Which service(s) is this request for? Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? In some cases our fargate workloads can hit 1024 packets/sec limit for network interface in VPC Resolver which affects stability of our services. DNS customisation for containers will allow to use own caching DNS forwarder to a) prevent such limits to hit; b) control all outbound dns requests for out services which increases observability; c) connect internal dns for hybrid infrastructures.

Are you currently working around this issue? Currently we found zero solutions to overcome this problem

[chrisforrette]

Big +1 on this!

[Vlaaaaaaad]

While chatting with @chrisforrette about this, my initial idea for a temporary workaround was to maybe run a sidecar container that caches DNS locally. Similar to how in Kubernetes world NodeLocalDNS caches DNS on each EC2 node.

That does not look possible due to dnsServers option not working with the awsvpc networking mode which is what Faragte uses :disappointed: Does anybody know if there's a way to force something like this?

[mreferre]

As pointed out some of the functionalities do not work when using the awsvpc networking mode (which is what Fargate uses). Not related to this exact problem but the way I worked around this limitation in the past was to do something like this.

Currently the ECS / Compose integration uses a sidecar mechanism to change the DNS configuration (to avoid having to change the application deployment code). There is work we are doing to try to lift some of these limitations.

[fabio-oliveira-efeo]

I managed to get around the problem using Route 53, but as it is a paid service, it would be interesting to be able to at least point to an external DNS server or edit /etc/hosts as in EC2 instances, by the container form itself.

[rikkei-manhnd]

+1 same problem here. I can't add DNS cache container to improve DNS efficiency

[kaypeter87]

+1 would love to see this

[danielebra]

+1

[KhitrovMaksim]

+1

[patradinesh]

+1

[marco-hoyer]

Hi team. This request has quite a lot of upvotes. I'm wondering if there are any news regarding it being implemented?

[vtstanescu]

As pointed out some of the functionalities do not work when using the awsvpc networking mode (which is what Fargate uses). Not related to this exact problem but the way I worked around this limitation in the past was to do something like this.

Currently the ECS / Compose integration uses a sidecar mechanism to change the DNS configuration (to avoid having to change the application deployment code). There is work we are doing to try to lift some of these limitations.

@mreferre Is there any updates on this feature request? If it's at least considered to be implemented? The workaround above implies running your container as root, since /etc/resolv.conf is 644. The sidecar option that Docker compose for ECS uses, seems to limit that by providing root just to the sidecar that modifies /etc/resolv.conf and then exits.

[mreferre]

@vtstanescu I moved to a different team and I am no longer working on containers/Fargate but I'd say the solution to this problem is to use ECS Service Connect which has since be released and allows customers to not having to worry about DIY sidecars or startup scripts that modify the hosts file. It does use a sidecar that is fully managed by AWS though.

I have tried it with the app where I applied my workaround (link in my first post) and Service Connect worked just fine (without the workaround in the startup script).

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect.html

[vtstanescu]

That is a service mesh solution for services hosted in ECS, right? What if you have a service that connects to third parties? Depending on the scale of that service you'll want to do some DNS caching to:

1. improve DNS response times
2. avoid hitting the 1024 pps/ENI to Amazon-provided DNS

[mreferre]

I did not have a chance to look into ECS Service Connect at this level of details. DNS caching is always a double edge sword: set it to be aggressive and you'll create ton of chattery, set it too loose and it will take hours to re-converge. Best would be to either engage support for your doubts on caching or try to ask here: https://repost.aws/.

[neilmillard]

Because docker mount host resolv.conf file inside container during start. In case of ECS it is the pause container which has a resolv.conf file mounted at /var/lib/docker/containers/{pause_container_id}/resolv.conf, This same file is mounted in all the containers. So if we change this file in one container, changes will get reflected in other containers as well of the same task.

[efagerho]

I ran into the same issue a few days ago with an application. My hackish workaround was to write a shared library that I can load with LD_PRELOAD and which intercepts and caches calls to getaddrinfo() for a few seconds. The set of addresses my application is resolving is quite small, so even caching DNS responses for a few seconds is enough to avoid hitting the DNS resolver limit.

https://github.com/efagerho/libnscache