Unauthorized error on public ECR repository

[jwitko]

When attempting to check image tags against the public ECR repository I am met with the following error:

failed to check container image \"opensearch\": failed to get tags from remote registry for "public.ecr.aws/opensearchproject/opensearch\": {\"errors\":[{\"code\":\"DENIED\",\"message\":\"Not Authorized\"}]}\n, requeuing

[jwitko]

A response from AWS support on this topic:

Hello ***,

*** here from AWS support! It was nice speaking with you over the call today. Please find below a quick recap of our discussion.

You reached out to us as you are unable to describe images from the AWS owned EKS private ECR repository "***.dkr.ecr.us-west-2.amazonaws.com".

----
failed to describe images: AccessDeniedException: User: *** is not authorized to perform: ecr:DescribeImages on resource: arn:aws:ecr:us-west-2:***:repository/eks/aws-ebs-csi-driver because no resource-based policy allows the ecr:DescribeImages action
----

In order to assist you better, I tried to replicate use-case in my internal account. I logged into the AWS Public ECR repository "***.dkr.ecr.us-west-2.amazonaws.com". After that I tried to describe the images from the repository and got the similar errors as shown below:

----
aws  ecr  describe-images --registry-id *** --repository-name ***.dkr.ecr.us-west-2.amazonaws.com --region us-west-2

An error occurred (AccessDeniedException) when calling the DescribeImages operation: User: arn:aws:sts::***:assumed-role/Admin is not authorized to perform: ecr:DescribeImages on resource: arn:aws:ecr:us-west-2:***:repository/***.dkr.ecr.us-west-2.amazonaws.com because no resource-based policy allows the ecr:DescribeImages action
----

I further tried to pull the images from this repository and I was able to download the image successfully. 

Therefore, I checked internally and found that users do not have access/permissions to list/query the Amazon's ECR repositories ( for example:- ***.dkr.ecr.us-west-2.amazonaws.com). They only have pull access to download the images from the Amazon's ECR repositories.

However, If you still would like to get the detailed information about the Amazon's ECR images, then You first have to download that image and push the same image to your own ECR repositories as shown below:-

----
1. $ docker tag ***.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller:v2.4.0 <tag>

2. $ docker push <image>

3. $ aws ecr describe-images --registry-id <repository> --image-ids imageTag=controller-v2.4.0 --region us-west-2

{
    "imageDetails": [
        {
            "registryId": "",
            "repositoryName": "",
            "imageDigest": "sha256:618bf3158323a641e4e6a79d3879dd8439776a4d8fdbb32ba8c1d6c3295c582e",
            "imageTags": [
                "controller-v2.4.0"
            ],
            "imageSizeInBytes": 18961873,
            "imagePushedAt": 1655874766.0,
----

I hope the above information will be helpful. Should you have any further concerns or issue regarding this case feel free to let me know I'll be more than happy to help you. I will keep this case in Pending for now and will set it to resolve by Monday.

Thank you and have a great day ahead!

We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.

Best regards,
***
Amazon Web Services

So the issue here is on the AWS side. Since we can't list images there likely isn't a simple way to make this work.
I'll close this issue since its not something wrong with this app.

[silazare]

Faced the same issue and seems it was fixed in https://github.com/aws/containers-roadmap/issues/1262 I've tried with AmazonEC2ContainerRegistryReadOnly policy attached to EKS node and from node itself we can list tags with token, but from app side the same policy doesn't work with Not Authorized error:

[root@ip-192-168-246-14 /]# TOKEN=$(curl -k https://public.ecr.aws/token/ | jq -r '.token')
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1625  100  1625    0     0   5811      0 --:--:-- --:--:-- --:--:--  5803
[root@ip-192-168-246-14 /]# curl -k -H "Authorization: Bearer $TOKEN" https://public.ecr.aws/v2/eks/aws-load-balancer-controller/tags/list | jq . | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1353  100  1353    0     0   3822      0 --:--:-- --:--:-- --:--:--  3832
{
  "name": "eks/aws-load-balancer-controller",
  "tags": [
    "v2.4.1",
    "v2.5.4-linux_amd64",
    "v2.4.7",
    "v2.3.0-linux_amd64",
    "v2.5.1",
    "v2.5.2-linux_arm64",
    "v2.5.4",

I assume that it needs to fetch the token for public.ecr.aws in the ECR client.

[jwitko]

@davidcollom I've reopened this issue since @silazare has pointed out that this is now something that sounds like it could be implemented with minor work

[davidcollom]

Hey @jwitko / @silazare Is this still an issue? We made some significant updates to the ECR logic/libraries in #182 and I'd like to know if this is resolved.

I don't have an AWS account/environment to hand to validate this, so would appreciate if it is possible that v0.5.5 and above to have this resolved 🙏

[silazare]

Hey @davidcollom thanks for the update!

I've checked today on one of my clusters and upgrade from 0.5.3 > 0.6.0, but unfortunately still the same issue with Public ECR gallery non authorized:

│ time="2024-07-04T05:30:04Z" level=error msg="error syncing 'aws-load-balancer-controller-76865d6bf6-26nzz/kube-system': failed to sync pod aws-load-balancer-controller-76865d6bf6-26nzz/kube-system: fa │
│ iled to check container image \"aws-load-balancer-controller\": failed to get tags from remote registry for \"public.ecr.aws/eks/aws-load-balancer-controller\": {\"errors\":[{\"code\":\"DENIED\",\"mes │
│ sage\":\"Not Authorized\"}]}\n, requeuing" module=controller

Image version is: image: public.ecr.aws/eks/aws-load-balancer-controller:v2.7.2