containers-roadmap
containers-roadmap copied to clipboard
aws
Fargate tasks: ability to set a proxy to be used
Summary
Feature request for Fargate tasks: ability to set a proxy to be used for pulling Docker images and automatically set the proxy env variables inside containers (as per https://docs.docker.com/network/proxy/#configure-the-docker-client)
Description
In environments where security is paramount it is very common to have to use a proxy for all outbound Internet traffic. Given AWS ECR currently doesn't offer a VPC endpoint, pulling images from it requires an Internet connection. It's possible to set such a proxy configuration using EC2 ECS tasks, but not when running Fargate tasks (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html#fargate-tasks-services-networking). Also of great usefulness would be if this proxy configuration would automatically propagate into launched containers of a Fargate task.
EDIT: There's now support for ECR on PrivateLink: https://aws.amazon.com/about-aws/whats-new/2019/01/aws-fargate--amazon-ecs--and-amazon-ecr-now-have-support-for-aws/, so this issue is only relevant for people wishing to use private repositories other than ECR.
#metoo We are unable to use Fargate without either a VPC endpoint for ECR (and presumably IAM) or without the ability to specify proxy for image download.
One of the solutions is to replace NAT instance with squid configured as transparent proxy. On squid instance you need to disable destination check and set it as destination in routing table.
You can read about it here: https://aws.amazon.com/blogs/security/how-to-add-dns-filtering-to-your-nat-instance-with-squid/
Also, Amazon has a blog post on how to make a highly available solution for NAT/Squid: https://aws.amazon.com/articles/high-availability-for-amazon-vpc-nat-instances-an-example/
Right now we have squid instance with ECR access allowed only. We run separate Fargate services squid/proxy for the rest of our needs (applications, etc.).
I agree, this is causing a problem for me too, but in my case my proxy is on a non-cloud part of the network, so routing everything through the proxy leads to unnecessary network traffic. So, while the ability to set a proxy would solve the problem functionally, I'm not sure it's the best solution.
My personal preferred options, in descending order of preference:
- Ability to run ECRs within the VPC - it's feels weird to me that my containers are living outside my VPC anyway, I'd expect the registry to be addressable in my private network somewhere.
- An endpoint like the s3 access endpoint
- Ability to set the proxy
+1 for "Why isn't Fargate still ready for this?" Not all the customers really love to use Public subnets in all of their VPCs...
I am not sure if i am missing somethings in my configuration, but i am able to get image pulled from ECR even when ENI have only private IP and Auto-assign public IP DISABLED mode. i am using same AWS account and region for both ECS service and ECR.
Since my containers have no reason to be reached out from the outside world, they are deployed in private subnets with an available route to a NAT gateway and i am able to pull the image from ECR.
@hridyeshpant I think having a NAT gateway allows you to have connectivity to the internet. Which basically means that all the apps on that subnet can reach the internet unrestricted (that's what the proxy server is used for). Which a lot of teams will not want.
now that https://aws.amazon.com/about-aws/whats-new/2019/01/aws-fargate--amazon-ecs--and-amazon-ecr-now-have-support-for-aws/ has been released, the need for this issue would only relate to using private repositories other than ECR
Yes, I agree with @ricoli . This is how we solved the problem for us. We have our Fargate tasks behind an HTTP proxy (without direct NAT access) that filters outgoing traffic just fine. Docker images are pulled via endpoint from ECR.
I really need this functionality too.
Meanwhile, we could use https://aws.amazon.com/about-aws/whats-new/2019/02/aws-fargate-now-has-support-for-aws-privatelink/
so this issue is only relevant for people wishing to use private repositories other than ECR.
Correct me if I'm mistaken but this also prevents pulling images from the public Docker Registry.
so this issue is only relevant for people wishing to use private repositories other than ECR.
Correct me if I'm mistaken but this also prevents pulling images from the public Docker Registry.
Only if said images are private of course
so this issue is only relevant for people wishing to use private repositories other than ECR.
Correct me if I'm mistaken but this also prevents pulling images from the public Docker Registry.
Only if said images are private of course
it's for (all) enterprise customers
still need this. Some customers do not use endpoints (for cost reasons among others) and prefer to treat AWS like a remote office with all external traffic going via their internal proxies
Our organisation also requires this functionality in order to use Fargate
Our company need this feature. Without a proxy option, we cant use fargate.
My use case is related to caching. We have the same image running many times on ECS and Fargate. At the moment every task downloads the same image separately. This results in unnecessary data transfer from NAT Gateway. Instead there could be a proxy inside the private subnet caching the image for subsequent requests.
My use case is related to caching. We have the same image running many times on ECS and Fargate. At the moment every task downloads the same image separately. This results in unnecessary data transfer from NAT Gateway. Instead there could be a proxy inside the private subnet caching the image for subsequent requests.
@raehalme wouldn't this use case be implemented using an ECR VPC endpoint?
My use case is related to caching. We have the same image running many times on ECS and Fargate. At the moment every task downloads the same image separately. This results in unnecessary data transfer from NAT Gateway. Instead there could be a proxy inside the private subnet caching the image for subsequent requests.
@raehalme wouldn't this use case be implemented using an ECR VPC endpoint?
@mreferre As VPC endpoint for ECR also charges for the data transfer I'm not sure I understand how it helps solve my use case.
You are correct. I have interpreted your request as a raw "how do I avoid using NAT GW" and mentioned VPC endpoint as a way to mitigate (not eliminate admittedly, as you noted) those costs.
My use case is related to caching. We have the same image running many times on ECS and Fargate. At the moment every task downloads the same image separately. This results in unnecessary data transfer from NAT Gateway. Instead there could be a proxy inside the private subnet caching the image for subsequent requests.
@raehalme wouldn't this use case be implemented using an ECR VPC endpoint?
@mreferre As VPC endpoint for ECR also charges for the data transfer I'm not sure I understand how it helps solve my use case.
@raehalme It occurred to me to note someone else is pointing out in a parallel issue that if you configure the GW endpoint S3 transfers cost $0. The docs actually seem to suggest to configure the S3 GW endpoint:
Amazon ECS tasks hosted on Fargate using platform version 1.4.0 or later require both the com.amazonaws.region.ecr.dkr and com.amazonaws.region.ecr.api Amazon ECR VPC endpoints as well as the Amazon S3 gateway endpoint to take advantage of this feature.
Given all major traffic goes through S3 (and ECR traffic is only minimal) you are basically not paying for data transfer. Would that be acceptable?