containers-roadmap
containers-roadmap copied to clipboard
aws
[Lambda] [request]: Cross-region cross-account ECR images
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Tell us about your request What do you want us to build? The possibility to use cross region cross account ECR images for Lambda.
Which service(s) is this request for? This could be for ECR and Lambda
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Back in the days if someone wants to use a Lambda functions in cross account and cross region the Lambda Zip have to be copied into that account and that region, consumeing S3 buckets per every region. What is the reason we have to do the same procedure for container images by replicating them into specific regions? The dockerhub isn't bounded to any AWS region so they can be used as well for Lambdas. The ECR private repository should be used for Lambdas with the freedom of cross region and cross account, without unecessary cost by the unwanted replications. So I want to use Lambdas with container images as the following workflow: (MyPrivate-ECR at eu-central-1 in Account A) --> (XYZ Lambda function at eu-west-1 in Account B)
@tomiszili Thanks for raising this. You can enable cross-region replication in your ECR registry today. This lets ECR automatically replicate your function images to another region, whether in the same account or a different one. One of the reasons we did this ECR feature is because you get the lowest latency when you pull in-region vs cross. Is there a reason in your use case where a cross-region pull for a Lambda function is as good as an in-region one?
@omieomye In our case the low latancy is not critical because this script only does some aws automations, based on resource creation/deletion events. The low cost is more important than the low latency for us, and it is a bit wasteful (both on cost and resource level) to replicate through all regions this repository, same for the s3 bucket case for the Lambda zip files.
@omieomye additionally the replication is a Push architecture, the source ECR needs to be aware of all the destinations it's replicating to.
There are cases where for various reasons you just can't change the configuration of the source ECR.
One such example I just ran into recently is using images from AWS marketplace Docker products for creating Lambda functions.
There's no easy way to clone the image from the Marketplace ECR to a local ECR straight from the Cloudformation template that creates the Lambda function. I'm currently trying to build something using CodeBuild and custom resources backed by another Lambda function but it's quite a complex and ugly solution for such a simple problem.
Ideally Lambda would just accept the original image and maintain a local ECR registry for each region where images would be cloned and used under the hood from it.
Similar reasons here, we're trying to architect an AWS setup where we have an "assets" account, which contains S3 buckets for lambda code, S3 buckets for static assets that are rolled into docker containers at build time, and now since lambda supports containers, ECR itself.
When you're programmatically spinning up a new AWS account on demand with CDK and deploying a simple stack (for arguments sake, three S3 buckets, one SQS queue and a single lambda) for a customer who demands isolation it's easier to use a pull model with permissions that make all accounts in an organisation unit permissible to pull containers into a lambda definition rather than setting up replication for what could turn out to be accounts added and removed every day.
Essentially quite like the idea of a fan out system than a push style system for maximum flexibility when defining a lambda function image.
Hello! Pulling Lambda images cross-account would be a great addition, please.
We are currently pulling ECR images, published to a single account eg. develop, from ECS and EKS clusters in multiple AWS accounts, eg. [ develop, staging, production ].
Instead of replicating ECR images to remote repositories for local consumption in another account, please consider allowing image pulls cross-account for Lambda, exactly like ECS and EKS, instead of dying on:
│ Error: error creating Lambda Function (1): InvalidParameterValueException: Image repository must be in the same account as the caller
│ {
│ RespMetadata: {
│ StatusCode: 400,
│ RequestID: "x"
│ },
│ Message_: "Image repository must be in the same account as the caller",
│ Type: "User"
│ }
Thank you!
As a workaround for AutoSpotting I've implemented a custom CloudFormation resource that copies the Docker image from a source ECR to local ECR usable from Lambda using the crane tool from https://github.com/google/go-containerregistry/. I'll publish it soon in a WIP branch.
Later edit: I also have equivalent code for Terraform at https://github.com/cloudutil/terraform-aws-autospotting/
Getting this to work would be great, as it would simplify a lot of my systems. For a lot of workloads the latency doesn't matter and can be mitigated using replication for the workloads where it is a big deal.
Part of the problem here is also that I only have the choice to replicate the entire registry- I can't just replicate the repositories I want. The bulk of my containers are for ECS, not Lambda, and therefore don't need to be replicated- my only options are to replicate them anyways or separate the lambda containers into a separate account to replicate only those. On top of that replication isn't a mirror, it's a copy, which means that each of these new registries we're copying into have to have separate management to remove all of their no longer used images.
@tedivm the CloudFormation custom resource I mentioned in my previous comment only clones the image relevant to the current Lambda function into a dedicated ECR created in the current AWS account. It's significantly increasing the complexity of my deployment code for something that could be easily handled in the Lambda backend, but it works like a charm. Here's the gist of it in my CloudFormation template https://github.com/AutoSpotting/AutoSpotting/pull/466/files#diff-0c788608818615939f48f8f9ce56c747c2313cb2efc82243c9b1f80dcaaa236fR775-R869
Will AWS enable lambda created in one AWS account (e.g A) to use an image in an ECR in another AWS account (e.g B)?
Need cross-account cross-region as well due to how application is deployed to individual customers.
@cristim - I ended up cheating and using IAM to restrict which images were pushed (I only gave permission for lambda-* to get replicated so that's all that ended up moving).
I see the tag "Work in progress" for quite some time already, is there any expected delivery date for it?
I am facing the 250Mb hard limit and the only way to move forward is docker, but I also fall into the same situation described above, we have a central aws account hosting all our ECR images and not being able to consume from there is breaking one of our DevOps policy.
Is this done now? https://aws.amazon.com/blogs/compute/introducing-cross-account-amazon-ecr-access-for-aws-lambda/
Is this done now? https://aws.amazon.com/blogs/compute/introducing-cross-account-amazon-ecr-access-for-aws-lambda/
This works across accounts in the same region. But certain use cases need it also to support using images from other regions.
@cristim as workaround you could replicate the images to other regions in shared account.
Honestly it makes sense that it only works across accounts in the same region- for DR purposes replicated across regions is useful anyways.
This update makes my life so much easier- thank you so much AWS team!
@cristim as workaround you could replicate the images to other regions in shared account.
Unfortunately I don't have access to the source account (it's maintained by the AWS Marketplace team), I'll keep using my workaround until this is fully supported
As a CDK user, I use this package https://github.com/cdklabs/cdk-ecr-deployment, it can deploy a docker image to local ECR, then the local image can be used in Lambda functions like this:
import * as ecrDeploy from 'cdk-ecr-deployment'
// 1. Create repo form a
const yourRepo = new ecr.Repository(this, 'yourRepo', {
repositoryName: 'yourRepo',
removalPolicy: cdk.RemovalPolicy.DESTROY,
});
const ecrImage = new ecrDeploy.ECRDeployment(this, 'ecrImage', {
src: new ecrDeploy.DockerImageName('srcImageUri'),
dest: new ecrDeploy.DockerImageName('destImageUri'),
});
// 2. Use this image for lambda function
const imageFunc = new lambda.DockerImageFunction(
this,
'imageFunc',
{
code: lambda.DockerImageCode.fromEcr(
ecr.Repository.fromRepositoryName(this, 'repo', 'repoName')
),
timeout: cdk.Duration.seconds(15),
memorySize: 10240,
}
);
Here is an example: https://github.com/cdklabs/cdk-ecr-deployment/blob/main/test/integ.ecr-deployment.ts
It is as described, Supports cross account but the same AWS region, which i think it does not meet certain use case which needs access to other AWS region as well.
Hi - is there any further progress with this? Our DR use case also highlighted the need for this or a solution to mitigate. many thanks!
Any update on this? it doesnt make sense to replicate to all regions and maintain so many ECR repositories, permissions, scanning, lifecycle
Also need this for a similar use case. Any update would be appreciated.
