argocd-image-updater icon indicating copy to clipboard operation
argocd-image-updater copied to clipboard

Published 20 hours ago • verified publisher icon argoproj-labs

tagsortmode set to "none" fails when used with Amazon ECR

Open diranged opened this issue 4 years ago • 3 comments

Describe the bug Amazon ECR's /tags/list endpoint returns back the list of tags not sorted by date (it might be lexicographical). To get around this, we're trying to set tagsortmode: none to have the image updater make subsequent API calls to figure out when each particular blob was uploaded. When we do this, we get errors back from the AWS ECR endpoint throwing 403's on the calls:

time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-6dc596c"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-a13b52e"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-faf148f"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-435ae68"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-8b0251a"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-0da85a5"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-5e2d313"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-9dc110b"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-c7841fe"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:252116da9bb49eba91dbcd7874c20a36eccc432439eaad356c3026d6166a081b"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:d7123eb7a7121961948503f687efece8ea6775d52717ecebe13c064686b873dd"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:c99ed1523e8da73f3f22aee87b053e53a8502678dd74f96bb981fec0cbcdc9e0"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:c2145f571d8940d08a65bb525ec5570c393277e7064d29edcf6fd75f55b2b042"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:c2145f571d8940d08a65bb525ec5570c393277e7064d29edcf6fd75f55b2b042"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:129549edceb37fc64fe9d75f41340b36596ed65d93ea77a9a5d2c08e802caf5e"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:d7123eb7a7121961948503f687efece8ea6775d52717ecebe13c064686b873dd"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:c2145f571d8940d08a65bb525ec5570c393277e7064d29edcf6fd75f55b2b042"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:54a4ee3284c2c588bd2ad6bda905ee536bfde4e3dbcdb54c59fd595beaade704"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:04616d20fc3485406963893785af497862b3df5ad5aac9f93f218cc2234f1a1c"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:54a4ee3284c2c588bd2ad6bda905ee536bfde4e3dbcdb54c59fd595beaade704"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:2f1ca1c0340c449942bac1cf40f79a0cf330ecb6cc2ee623e4e6f5b0dc4334f0"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:8e8d999649ab7a60f514787083d340ca6cd8b76a563bde5e7629d6e188fbc401"
time="2021-06-15T22:30:43Z" level=error msg="error fetching metadata for myorg/myapp:main-cca0372: could not get metadata: Head \"https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:252116da9bb49eba91dbcd7874c20a36eccc432439eaad356c3026d6166a081b\": http: non-successful response (status=403 body=\"\")"
time="2021-06-15T22:30:43Z" level=trace msg="released semaphore and terminated waitgroup"
time="2021-06-15T22:30:43Z" level=trace msg="acquired metadata semaphore"
time="2021-06-15T22:30:43Z" level=trace msg="Getting manifest for image myorg/myapp:main-9bfbeef (operation 22/24)"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-de66320"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:0eb8a8b378010e73b2a7e81daab79d9a6fcad7ebe6fc5ca9f5cbe19a10c1452f"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:e02ccbfcfd0e1a65116d7f196ce00b073d2bf1256436b2df10e2f0e21e4c610f"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:0115d417b3566bafcd95a9d130e336bd7320069f11fb0a04d45d8457ec621415"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:54a4ee3284c2c588bd2ad6bda905ee536bfde4e3dbcdb54c59fd595beaade704"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:8c97f70618eccbb73831496e0142c528b27996964013695f883cd0c3cc711ba1"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:8d53a0eafb8613965ce1e3e085d0d19c53d6fab33ea310699b7e0a543a5c0c9c"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:81ec223defc03ce6c2b24165a7fbd520fc0b6ac168ffb7c81a82ad1a5676bd33"
time="2021-06-15T22:30:43Z" level=error msg="error fetching metadata for myorg/myapp:main-9f971e8: could not get metadata: Head \"https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:c99ed1523e8da73f3f22aee87b053e53a8502678dd74f96bb981fec0cbcdc9e0\": http: non-successful response (status=403 body=\"\")"
time="2021-06-15T22:30:43Z" level=trace msg="released semaphore and terminated waitgroup"
time="2021-06-15T22:30:43Z" level=trace msg="acquired metadata semaphore"
time="2021-06-15T22:30:43Z" level=trace msg="Getting manifest for image myorg/myapp:main-39c6706 (operation 23/24)"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-9bfbeef"
time="2021-06-15T22:30:43Z" level=error msg="error fetching metadata for myorg/myapp:main-9dc110b: could not get metadata: Head \"https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:129549edceb37fc64fe9d75f41340b36596ed65d93ea77a9a5d2c08e802caf5e\": http: non-successful response (status=403 body=\"\")"
time="2021-06-15T22:30:43Z" level=trace msg="released semaphore and terminated waitgroup"
time="2021-06-15T22:30:43Z" level=trace msg="acquired metadata semaphore"
time="2021-06-15T22:30:43Z" level=trace msg="Getting manifest for image myorg/myapp:main-7f2db5e (operation 24/24)"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-39c6706"
time="2021-06-15T22:30:43Z" level=error msg="error fetching metadata for myorg/myapp:main-8b0251a: could not get metadata: Head \"https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:8e8d999649ab7a60f514787083d340ca6cd8b76a563bde5e7629d6e188fbc401\": http: non-successful response (status=403 body=\"\")"
time="2021-06-15T22:30:43Z" level=trace msg="released semaphore and terminated waitgroup"
time="2021-06-15T22:30:43Z" level=trace msg="acquired metadata semaphore"

However, when we try CURL'ing the endpoint's reported in the URLs above, we can get the response just fine:

bash$ TOKEN=$(aws --region us-west-2 ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken')
$ curl --head -i -H "Authorization: Basic $TOKEN" https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:252116da9bb49eba91dbcd7874c20a36eccc432439eaad356c3026d6166a081b
HTTP/1.1 200 OK
Content-Length: 7394
Content-Type: application/vnd.docker.container.image.v1+json
Docker-Content-Digest: sha256:252116da9bb49eba91dbcd7874c20a36eccc432439eaad356c3026d6166a081b
Docker-Distribution-Api-Version: registry/2.0
Date: Tue, 15 Jun 2021 22:36:36 GMT

To Reproduce Follow the instructions I posted originally at https://github.com/argoproj-labs/argocd-image-updater/issues/112#issuecomment-776791750... then switch tagsortmode to none

Expected behavior I expect the API calls to work because I can execute them myself via Curl..

Version Latest master build as of https://github.com/argoproj-labs/argocd-image-updater/commit/1d19e6a2f5dc5658c4c6539cf8a4518a1233bb4e

diranged avatar Jun 15 '21 22:06 diranged

Unfortunately, I do not have an AWS account to test ECR functionality so I am not able to set up a reproduction scenario here.

I have a few questions, tho:

  1. Do you receive a 403 for every meta data request, or only for some?

  2. Do you use the same AWS account when using curl and the request succeeds? HTTP 403 is usually emitted if the request was authenticated successfully, but does not have permissions to access the requested resource.

  3. I'm not sure how ECR handles request limits, but do you have any quota set to your registry maybe for max pulls/second? Other registries will usually emit HTTP 429 if the limit is exceeded, but I'm unsure how this is with ECR. Keep in mind that the Image Updater uses concurrency extensively, so low limits may be reached quickly.

jannfis avatar Jun 16 '21 08:06 jannfis

@jannfis I just figured the issue out - and perhaps its time to start a document somewhere in the image updater docs talking about ECR. We were missing the ecr:BatchCheckLayerAvailability permission on our IAM Permissions Boundary. For anyone in the future finding this issue, here's the permissions that are now working with tagsortmode: none:

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability",
                "ecr:ListTagsForResource"
            ],
            "Resource": [
                "*"
            ],
            "Sid": "AmazonECRReadAccess"
        }
    ]
}

diranged avatar Jun 16 '21 13:06 diranged

hey @diranged how did you manage to get the awscli inside argocd-image-updater?

AndresJulia avatar Aug 13 '21 23:08 AndresJulia