Add support for namespaces in proxy registry

[DerEnderKeks]

This PR adds support for multiple remote registries when using distribution in proxy mode using the ns query parameter, which is proposed to the OCI spec here and already implemented in Containerd.

I implemented it in a mostly backwards compatible way, the old proxy config settings remain intact. When the new config options are not used, it behaves as before. When namespaces are enabled, all remote repositories are stored prepended with the domain of the remote. The remote is either determined using the ns query parameter, or when missing using the domain that was prepended to the image name.

Pulling an image the following ways would both pull the image some/image:latest from external.example.org:

• Pulling registry-proxy.example.com/some/image:latest, with ns set to external.example.org
• Pulling registry-proxy.example.com/external.example.org/some/image:latest

To enable the feature one would put the following in the config file:

proxy:
  enablenamespaces: true
  credentials: # optional
    'https://example.com':
      username: someuser
      password:  somepassword

I'd appreciate any feedback.

[milosgajdos]

You need to sign DCO as per the contributor's guidelines.

[codecov-commenter]

Codecov Report

Patch coverage: 7.05% and project coverage change: -0.31 :warning:

Comparison is base (e5d5810) 56.58% compared to head (aa3255a) 56.27%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3864      +/-   ##
==========================================
- Coverage   56.58%   56.27%   -0.31%     
==========================================
  Files         105      105              
  Lines       10636    10693      +57     
==========================================
  Hits         6018     6018              
- Misses       3946     4002      +56     
- Partials      672      673       +1     

Impacted Files	Coverage Δ
configuration/configuration.go	64.38% <ø> (ø)
registry/proxy/proxyauth.go	0.00% <0.00%> (ø)
registry/proxy/proxyregistry.go	0.00% <0.00%> (ø)
registry/handlers/app.go	47.01% <57.14%> (-0.20%)	:arrow_down:
registry/proxy/proxyblobstore.go	53.52% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[dcarrion87]

@DerEnderKeks this looks useful!

We're wanting to mirror multiple registries and just wondering if this would solve our problem. We run containerd like this:

[plugins.cri.registry]
  [plugins.cri.registry.mirrors]
    [plugins.cri.registry.mirrors."*"]
      endpoint = ["https://myregistry.mydomain.com"]

But as you know if say ghcr.io/myimage comes through it bypasses the mirror.

Just looking at the code it just seems like as long as ns= comes through and enablenamespaces: true this should just work?

Do you know if docker is looking to support this mechanism too?

[dcarrion87]

Well that just worked. This feature is great! Would love to see this merged @milosgajdos @DerEnderKeks!

[dcarrion87]

Just bumping this to see what is necessary to have this merged? This solves a few issues that have been raised in this project for multi remote.

[marco-svitol]

Would love to have this feature! We are using proxy registry to speed up our Kind cluster integration tests. As now we must create one proxy registry per remote registry. This feature would enale us to use a single proxy.

[dcarrion87]

@marco-svitol we ended up forking and publishing our own image as it's crickets on when this is going to be merged. It's been running for a week and so far so good. We've got some hefty images across dockerhub and ghcr going through it.

[marco-svitol]

@dcarrion87 is the image publicly available? I would like to test it too. thanks!

[dcarrion87]

We've currently got it published here: https://github.com/harrison-ai/cobalt-distribution/pkgs/container/cobalt-distribution

We'll eventually bring it down once it's merged into this project so best not to rely on it.

[milosgajdos]

This is building support for an OCI proposal that has not been merged. I'd encourage you to ask OCI maintainers on the OCI PR proposal to consider merging it. We are not against adding this in.

[dcarrion87]

I bumped https://github.com/opencontainers/distribution-spec/pull/66

[hoozecn]

seems not working for upstream with anonymous credential like docker.io

[DerEnderKeks]

Works fine for me. I tested with docker.io, quay.io and probably some others. You got to be a little more specific in what you mean with "not working". Any related log output? What exactly doesn't work?

[hoozecn]

Works fine for me. I tested with docker.io, quay.io and probably some others. You got to be a little more specific in what you mean with "not working". Any related log output? What exactly doesn't work?

Here was how I tested:

1. git clone the latest version of https://github.com/distribution/distribution
2. merge this PR
3. rungo run ./cmd/registry serve config.yaml

config.yaml

version: 0.1
log:
  accesslog:
    disabled: false
  level: debug
  formatter: json
loglevel: debug
storage:
  filesystem:
    rootdirectory: /tmp/registry
    maxthreads: 100
http:
  addr: localhost:5000
proxy:
  enablenamespaces: true

4. run docker pull localhost:5000/docker.io/library/busybox:latest

got an error response:

Error response from daemon: manifest for localhost:5000/docker.io/library/busybox:latest not found: manifest unknown: manifest unknown

[hoozecn]

@DerEnderKeks I tried to fix the problem and it seems to work with the patch

https://github.com/DerEnderKeks/distribution/pull/1/files

[DerEnderKeks]

The issue only occurred when pulling from docker.io and with the domain in the path, not when using the ns parameter, because I previously extracted the remote URL after setting up the credentials. This is now fixed.

@hoozecn your PR pointed me to the right place, thanks.

[dtomasi]

Is there any progress? This feature would be extremely useful for me.

[Jamstah]

@DerEnderKeks Can you please rebase this?

One quick review comment, there appear to be no docs changes here, which there should be with configuration format changes. When that's done, can we please indicate in some way that the feature is based on a spec proposal and subject to change?

[darzanebor]

greate feature, any updates when it would be merged?

[DerEnderKeks]

I rebased my branch and added documentation for the new config options. As requested, I specificly mentioned that this feature is based on a proposal.

Is there anything else required to get this merged?

[darzanebor]

@DerEnderKeks could you please fix issue "Clear-text logging of sensitive information"? I think it's the final issue which blocks this feature.

[DerEnderKeks]

I don't quite understand why CodeQL thinks that the password is logged. The password is never passed to any logging call and the check only failed in places I didn't even modify. Maybe a false positive?

[darzanebor]

@milosgajdos could you please advise is it really false positive in job CodeQL?

[milosgajdos]

We've had issues with CodeQL acting silly in the past and when the GH team introduced some bugs. Not quite sure what's going on here tbqh.

[darzanebor]

@marco-svitol @Jamstah could you please merge it?

[milosgajdos]

@marco-svitol @Jamstah could you please merge it?

Before anything is merged we'll need the referenced distribution-spec proposal to be merged (https://github.com/opencontainers/distribution-spec/pull/66)

[marco-svitol]

@marco-svitol @Jamstah could you please merge it?

I'm not in charge of this nor I have the authorization.