Support public.ecr.aws

[oleksandr-gubchenko]

Support for public ECR seems to be missing for docker-registry type.

[jurre]

Can you show the config you're using and the error that you're getting?

[oleksandr-gubchenko]

@jurre conf:

registries:
  ecr-docker:
    type: docker-registry
    url: https://public.ecr.aws
    username: ${{secrets.ECR_AWS_ACCESS_KEY_ID}}
    password: ${{secrets.ECR_AWS_SECRET_ACCESS_KEY}}
updates:
- package-ecosystem: docker
  directory: "/"
  registries:
    - ecr-docker
  schedule:
    interval: "daily"

trace:

  proxy | 2021/09/09 10:39:04 [012] * authenticating docker registry request (host: public.ecr.aws)
  proxy | 2021/09/09 10:39:05 [012] 400 https://public.ecr.aws:443/v2/prima/elixir/tags/list
updater | I, [2021-09-09T10:39:05.084277 #9]  INFO -- sentry: ** [Raven] Sending event 11d6b5df90424fd2a02c0dc4ae1a251e to Sentry
  proxy | 2021/09/09 10:39:05 [014] POST https://sentry.io:443/api/1451818/store/
  proxy | 2021/09/09 10:39:05 [014] 200 https://sentry.io:443/api/1451818/store/
updater | ERROR <job_200928659> Error processing prima/elixir (RestClient::BadRequest)
updater | ERROR <job_200928659> 400 Bad Request
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
updater | ERROR <job_200928659> /usr/lib/ruby/2.7.0/net/http.rb:933:in `start'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/docker_registry2-1.10.0/lib/registry/registry.rb:251:in `doreq'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/docker_registry2-1.10.0/lib/registry/registry.rb:26:in `doget'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/docker_registry2-1.10.0/lib/registry/registry.rb:65:in `tags'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-docker-0.162.0/lib/dependabot/docker/update_checker.rb:231:in `tags_from_registry'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-docker-0.162.0/lib/dependabot/docker/update_checker.rb:173:in `comparable_tags_from_registry'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-docker-0.162.0/lib/dependabot/docker/update_checker.rb:148:in `fetch_latest_version'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-docker-0.162.0/lib/dependabot/docker/update_checker.rb:60:in `latest_version'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:460:in `all_versions_ignored?'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:191:in `check_and_create_pull_request'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:80:in `check_and_create_pr_with_error_handling'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `block in run'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `each'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `run'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'
updater | ERROR <job_200928659> /home/dependabot/dependabot-updater/lib/dependabot/base_job.rb:28:in `run'
updater | ERROR <job_200928659> bin/update_files.rb:21:in `<main>'
updater | INFO <job_200928659> Finished job processing
updater | time="2021-09-09T10:39:05Z" level=info msg="task complete" container_id=job-200928659-updater exit_code=0 job_id=200928659 step=updater```

[jurre]

Thanks!

Typically ECR registry URLs look something like this: 123456789123.dkr.ecr.us-east-2.amazonaws.com, it looks like our auth proxy is trying to authenticate to this registry as a regular docker registry (using OAuth). ECR uses basic auth instead, and ot looks like we may also need to support this endpoint for basic auth?

By the way, it seems like https://public.ecr.aws redirects to https://gallery.ecr.aws/, which is not likely the issue here, but would we need to support both?

[oleksandr-gubchenko]

The endpoint for login/pull etc is public.ecr.aws as you can see here for example: https://gallery.ecr.aws/amazonlinux/amazonlinux

[jurre]

It's a little more involved than simply adding that URL, auth against this registry is slightly different from others so it might take a minute before we'll be able to pick this up, but I'll try to keep you posted on progress here, thanks for reporting this 🙇 👍

[lorengordon]

Just curious before I actually try it, but if the ECR public registry is public, does dependabot already send updates for images that do not require authentication, when just specifying (e.g.) FROM public.ecr.aws/codebuild/amazonlinux2-x86_64-standard:3.0 in the Dockerfile, without the registries key in the dependabot config? I know it does for images on quay.io...

[oleksandr-gubchenko]

Just curious before I actually try it, but if the ECR public registry is public, does dependabot already send updates for images that do not require authentication, when just specifying (e.g.) FROM public.ecr.aws/codebuild/amazonlinux2-x86_64-standard:3.0 in the Dockerfile, without the registries key in the dependabot config? I know it does for images on quay.io...

It doesn't work without authentication, dunno why

[lorengordon]

That's really weird. I am able to pull the image with no authentication. @jurre maybe an easier approach here would be to remove the authentication requirement for this endpoint, make it optional?

I have no AWS credentials in the env, no AWS_PROFILE set, and no default config:

❯ docker pull public.ecr.aws/codebuild/amazonlinux2-x86_64-standard:3.0
3.0: Pulling from codebuild/amazonlinux2-x86_64-standard
430235fb155c: Pulling fs layer
8a9e14dfc60e: Pulling fs layer
f9e77b87f14a: Pulling fs layer
5c738dd5dde5: Waiting
9460b27665a1: Waiting
...

[oleksandr-gubchenko]

I think dependabot is trying to list tags for that image from endpoint like: https://public.ecr.aws/codebuild And get unauthorized error, also it may depend on repository type (aws verified or just a random one...)

[lorengordon]

Oh right on. Just found the related issue and feature request, apparently ECR Public doesn't support the list-image-tags API of the docker registry spec. 🤦‍♂️

https://github.com/aws/containers-roadmap/issues/1262

[rr-nick-tan]

seems issue, seems dependabot failed to list the tags from aws public ecr

updater | INFO <job_271798525> Checking if docker/library/ubuntu 20.04 needs updating
  proxy | 2022/01/26 18:46:48 [036] GET https://public.ecr.aws:443/v2/docker/library/ubuntu/tags/list
  proxy | 2022/01/26 18:46:48 [036] 401 https://public.ecr.aws:443/v2/docker/library/ubuntu/tags/list
  proxy | 2022/01/26 18:46:48 [038] GET https://public.ecr.aws:443/token/?service=public.ecr.aws&scope=aws
  proxy | 2022/01/26 18:46:48 [038] 200 https://public.ecr.aws:443/token/?service=public.ecr.aws&scope=aws
  proxy | 2022/01/26 18:46:48 [040] GET https://public.ecr.aws:443/v2/docker/library/ubuntu/tags/list
  proxy | 2022/01/26 18:46:48 [040] 404 https://public.ecr.aws:443/v2/docker/library/ubuntu/tags/list
  proxy | 2022/01/26 18:46:48 [042] GET https://public.ecr.aws:443/v2/docker/library/ubuntu/tags/list
  proxy | 2022/01/26 18:46:48 [042] 401 https://public.ecr.aws:443/v2/docker/library/ubuntu/tags/list
  proxy | 2022/01/26 18:46:48 [044] GET https://public.ecr.aws:443/token/?service=public.ecr.aws&scope=aws
  proxy | 2022/01/26 18:46:48 [044] 200 https://public.ecr.aws:443/token/?service=public.ecr.aws&scope=aws
  proxy | 2022/01/26 18:46:48 [046] GET https://public.ecr.aws:443/v2/docker/library/ubuntu/tags/list
  proxy | 2022/01/26 18:46:48 [046] 404 https://public.ecr.aws:443/v2/docker/library/ubuntu/tags/list
  proxy | 2022/01/26 18:46:48 [048] GET https://public.ecr.aws:443/v2/docker/library/ubuntu/tags/list
  proxy | 2022/01/26 18:46:48 [048] 401 https://public.ecr.aws:443/v2/docker/library/ubuntu/tags/list
  proxy | 2022/01/26 18:46:48 [050] GET https://public.ecr.aws:443/token/?service=public.ecr.aws&scope=aws
  proxy | 2022/01/26 18:46:48 [050] 200 https://public.ecr.aws:443/token/?service=public.ecr.aws&scope=aws
  proxy | 2022/01/26 18:46:48 [052] GET https://public.ecr.aws:443/v2/docker/library/ubuntu/tags/list
  proxy | 2022/01/26 18:46:48 [052] 404 https://public.ecr.aws:443/v2/docker/library/ubuntu/tags/list
updater | I, [2022-01-26T18:46:48.987026 #8]  INFO -- sentry: ** [Raven] Sending event c9bf901b932148a5b54accfc01aca7d0 to Sentry
  proxy | 2022/01/26 18:46:49 [054] POST https://sentry.io:443/api/1451818/store/
  proxy | 2022/01/26 18:46:49 [054] 200 https://sentry.io:443/api/1451818/store/
updater | ERROR <job_271798525> Error processing docker/library/ubuntu (DockerRegistry2::NotFound)
updater | ERROR <job_271798525> 404 Not Found
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/docker_registry2-1.10.0/lib/registry/registry.rb:327:in `rescue in do_bearer_req'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/docker_registry2-1.10.0/lib/registry/registry.rb:307:in `do_bearer_req'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/docker_registry2-1.10.0/lib/registry/registry.rb:269:in `rescue in doreq'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/docker_registry2-1.10.0/lib/registry/registry.rb:245:in `doreq'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/docker_registry2-1.10.0/lib/registry/registry.rb:26:in `doget'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/docker_registry2-1.10.0/lib/registry/registry.rb:65:in `tags'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-docker-0.171.1/lib/dependabot/docker/update_checker.rb:231:in `tags_from_registry'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-docker-0.171.1/lib/dependabot/docker/update_checker.rb:173:in `comparable_tags_from_registry'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-docker-0.171.1/lib/dependabot/docker/update_checker.rb:148:in `fetch_latest_version'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/vendor/ruby/2.7.0/gems/dependabot-docker-0.171.1/lib/dependabot/docker/update_checker.rb:60:in `latest_version'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:464:in `all_versions_ignored?'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:193:in `check_and_create_pull_request'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `check_and_create_pr_with_error_handling'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `block in run'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `each'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:56:in `run'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'
updater | ERROR <job_271798525> /home/dependabot/dependabot-updater/lib/dependabot/base_job.rb:35:in `run'
updater | ERROR <job_271798525> bin/update_files.rb:22:in `<main>'
updater | INFO <job_271798525> Finished job processing

[lorengordon]

Might be worth checking this again. ECR just released support for listing tags on public repos... https://github.com/aws/containers-roadmap/issues/1262#issuecomment-1196017565

[jalaziz]

Might be worth checking this again. ECR just released support for listing tags on public repos... aws/containers-roadmap#1262 (comment)

Won't work because AWS completely ignored the OCI spec and implemented their own response format 🤦🏼

[jlbutler]

Hi all,

I have made a comment on our issue tracking tags missing from ECR Public's v2 API.

The quick ask for feedback: if we iterate toward full OCI compliance, for the Dependabot use case is it OK for us to consider deferring the lexical sorting and transparent pagination token aspect of the spec, to focus on a first delivery that is the correct response body? We're very interested in unblocking use cases like this primarily.

@jurre and others, if you have the time, can you reply here or on the roadmap issue? And, thanks for your time!

[jurre]

@jlbutler it's been a minute since I dug into this, I'm reaching out to some team members to see if they have more context on this, and otherwise will try to dig into it a bit next week, but happy to hear about these plans 🎉

[bdragon]

@jbutler thanks for the transparency.

I don't think it's worth postponing an iteration that returns a spec-compliant response body in order to implement tag pagination.

It looks like the client used in dependabot-core for listing tags supports pagination of tags per the OCI spec, but doesn't require its usage. If the response format becomes compliant, I think it "should just work" and when support for pagination of tags is added, we'll be able to leverage that if needed.

Without lexical sorting or support for pagination, my only questions are:

• will you impose any limit on the number of tags returned on the ECR side?
• if tags are not lexically sorted, is there any rhyme or reason to the order in which they'll be returned? e.g., latest tags first?

[jlbutler]

thanks @bdragon and @jurre for the response!

Thanks for the feedback, it aligns unanimously so far with other projects and customers thinking we're best to start with the body and iterate. Great!

On sorting, tags are currently sorted by an internal element, so by all measure they are unsorted - though the order is consistent for a given set of tags.

On limit, this is something we're discussing internally. The spec doesn't really allow for it, but might be something we bring to OCI distribution for discussion. Presumably with enough tags, we could hit natural timeouts or unexpectedly large response body.

Please let me know if either of those responses seem troublesome. And thanks again!

[jalaziz]

On sorting, tags are currently sorted by an internal element, so by all measure they are unsorted - though the order is consistent for a given set of tags.

Without predictable sorting or pagination, how is dependabot supposed to know there's actually a new version of an image available? Unless the plan is to return all tags in one potentially giant page?

The Docker updater seems to sort in memory, but as far as I can tell, it requires all the tags to be available to choose the correct latest version.

[jlbutler]

On sorting, tags are currently sorted by an internal element, so by all measure they are unsorted - though the order is consistent for a given set of tags.

Without predictable sorting or pagination, how is dependabot supposed to know there's actually a new version of an image available? Unless the plan is to return all tags in one potentially giant page?

The Docker updater seems to sort in memory, but as far as I can tell, it requires all the tags to be available to choose the correct latest version.

Sorry missed this earlier. This is in reference to lexical sorting, tags aren't sorted by date or by any means you may consider something latest versions in any implementation that I'm aware of. Mostly we were concerned w/ sorting as relates to pagination, where a transparent tag is used. Non-lexically sorted results with an opaque token for pagination should work ok for this use case. But, let me know if not, thanks!

[elgohr]

Regarding the 401 at the list endpoint: https://github.com/aws/containers-roadmap/issues/1806#issuecomment-1228804975

[bdragon]

Thanks @jlbutler

I just want to check my understanding. For a first iteration, the response body will be OCI-compliant, lexical sorting will not be implemented yet (although a given set of tags will be consistently ordered), and pagination of tags will be implemented, but using an opaque token for now (rather than a transparent token per the spec). Is that right?

[jeffwidman]

Looks like the tags endpoint is now live: https://github.com/aws/containers-roadmap/issues/1262#issuecomment-1255298989

I haven't followed this issue closely, so I'm unclear if we need to do something on our end, or if this will just work™.

Can anyone using ECR try it and report back?

[jeffwidman]

Based on https://github.com/aws/containers-roadmap/issues/1262#issuecomment-1256827771, Dependabot should "just work" for Dockerfiles pulling from AWS ECR! 🎉

Many thanks to @jlbutler and the rest of the AWS ECR team for doing the work to implement OCI compliant APIs.