Please provide ETA for supporting replication to Google Artifact Registry as Registry Endpoint Provider

[northernpat]

Google Cloud Platform still supports Google Container Registry but only as a legacy service and it offers Artifact Registry as a better alternative which has many features including artifact support and granular access control. We have a harbor registry as an internal development environment and have been able to replicate it to GCR without issues. We are unable to replicate to Artifact Registry since Google Artifact Registry is not listed as a Registry End Point Provider. Could you provide an ETA on when Support for Artifact Registry will be included in Harbor? Kind Regards Patrick

[zyyw]

could you please try Docker Registry for Artifact Registry?

[dmizin]

Same request. Tried "Docker Registry" but push to Google Artifact Registry fails with with following: http status code: 401, body: {"errors":[{"code":"UNAUTHORIZED","message":"not authenticated: No valid credential was supplied."}]} According to Google troubleshooting - this maybe an issue of "Check that your client, or any intermediate proxy, is not adding an Authorization header to requests to Cloud Storage. Any request with an Authorization header, even if empty, is validated as if it were an authentication attempt.". Registry access is setup with json key and when "Test Connection" is pressed - check is all GREEN.

Thank you Dmitry Mizin

[northernpat]

@zyyw , I have created the GAR registry as a remote registry and used the _json_key for the service account for the authentication. However, although the registry ping test is successful and the service owner has read and written permission, I am getting the following error.

"sha256:0ac33e5f5afa79e084075e8698a22d574816eea8d7b7d480586835657c3e1c8b(the 1th running)... 2022-06-30T23:21:28Z [ERROR] [/controller/replication/transfer/image/transfer.go:332]: failed to pushing the blob sha256:0ac33e5f5afa79e084075e8698a22d574816eea8d7b7d480586835657c3e1c8b, size 1472: http status code: 401, body: {"errors":[{"code":"UNAUTHORIZED","message":"not authenticated: No valid credential was supplied."}]} 2022-06-30T23:21:28Z [INFO] [/controller/replication/transfer/image/transfer.go:278]: will retry 2s later 2022-06-30T23:21:30Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:0ac33e5f5afa79e084075e8698a22d574816eea8d7b7d480586835657c3e1c8b(the 2th running)... 2022-06-30T23:21:34Z [ERROR] [/controller/replication/transfer/image/transfer.go:332]: failed to pushing the blob sha256:0ac33e5f5afa79e084075e8698a22d574816eea8d7b7d480586835657c3e1c8b, size 1472: http status code: 401, body: {"errors":[{"code":"UNAUTHORIZED","message":"not authenticated: No valid credential was supplied."}]} 2022-06-30T23:21:34Z [INFO] [/controller/replication/transfer/image/transfer.go:278]: will retry 4s later 2022-06-30T23:21:38Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:0ac33e5f5afa79e084075e8698a22d574816eea8d7b7d480586835657c3e1c8b(the 3th running)..."

[northernpat]

There seems to be an issue with passing the authorization key

[mrbuk]

@zyyw I have looked into this and believe to have identified the issue.

The native docker registry adapter adds the authentication header only to URLs that contain a /v2/. See the code

In case of Google Artifact Registry the actual upload endpoint does not contain a /v2 but instead /artifacts-uploads like showing below

18:30:35 HTTPS GET     europe-west1-docker.pkg.dev /v2/                                                                                                                  401   application/json
18:30:35 HTTPS HEAD    europe-west1-docker.pkg.dev /v2/artifact-registry-sync-121/sync-01/mrbuk/nginx/manifests/1.21                                                     404       [no content]
18:30:35 HTTPS HEAD    europe-west1-docker.pkg.dev /v2/artifact-registry-sync-121/sync-01/mrbuk/nginx/blobs/sha256:036ae8e42548a69ca1cd3f8862cc243dd90230258abecee936cd… 404       [no content]
18:30:35 HTTPS POST    europe-west1-docker.pkg.dev /v2/artifact-registry-sync-121/sync-01/mrbuk/nginx/blobs/uploads/                                                     202       [no content]
18:30:35 HTTPS PUT     europe-west1-docker.pkg.dev /artifacts-uploads/namespaces/artifact-registry-sync-121/repositories/sync-01/uploads/AB38jmJfA4_GYLVqot73sX7TCFk9AK… 401   application/json

To fix this I would see two options:

1. Rewrite the check (lines 118 - 121) so that it adds the Authorization header to non /v2/ URLs of the same host (different backends like S3 should have a different hostname). Not sure if this is a good option as it would alter existing behavior.
2. Implement a custom adapter for Google Artifact Registry that uses a different authorizer. I have implemented a adapter including some basic tests, tested it in my environment and it worked so far. If you are interested I can create a Pull Request.

[Vad1mo]

2. is the better option to explicitly support GAR, IMHO,

We welcome a PR supporting GAR.

[jonjohnsonjr]

1. Rewrite the check (lines 118 - 121) so that it adds the Authorization header to non /v2/ URLs of the same host

I believe this matches what all other clients do, but I can understand not wanting to alter existing behavior.

[lindsayismith]

Hi - Eng from Artifact Registry here. Thanks for looking into this, customers have asked about Harbor support. You are correct that uploads are redirected to the 'artifact-uploads' url path.

There is also the download side as well - does Harbor need to pull from AR as well? If so - you also need to include the 'artifacts-downloads' path as well.

is there anything we can help with to move this PR along? I would be happy to set up a project or repository to test against if that's useful.

[mrbuk]

@lindsayismith is looks like currently the proper header to /artifacts-downloads is added as a "side-effect". In contrast to /artifacts-uploads, /artifacts-downloads is done via a 302 redirect as observed:

20:06:55 HTTPS GET  …pe-west1-docker.pkg.dev /v2/                                                                                              401 …plication/json   96b  27ms
20:06:55 HTTPS GET  …pe-west1-docker.pkg.dev /v2/token?scope=registry%3Acatalog%3A%2A&service=                                                 200 …plication/json  1.7k  70ms
20:06:55 HTTPS GET  …pe-west1-docker.pkg.dev /v2/_catalog?n=1000                                                                               200 …plication/json  116b 330ms
20:06:56 HTTPS GET  …pe-west1-docker.pkg.dev /v2/artifact-registry-sync-34412/sync-01/alpine/tags/list                                         200 …plication/json  349b  76ms
20:06:56 HTTPS GET  …pe-west1-docker.pkg.dev /v2/                                                                                              401 …plication/json   96b  74ms
20:06:56 HTTPS GET  …pe-west1-docker.pkg.dev /v2/token?scope=repository%3Aartifact-registry-sync-34412%2Fsync-01%2Falpine%3Apull&service=      200 …plication/json  1.8k 121ms
20:06:56 HTTPS GET  …pe-west1-docker.pkg.dev /v2/artifact-registry-sync-34412/sync-01/alpine/manifests/3.15.5                                  200 …nifest.v2+json  528b  62ms
20:06:56 HTTPS GET  …pe-west1-docker.pkg.dev /v2/token?scope=repository%3Aartifact-registry-sync-34412%2Fsync-01%2Falpine%3Apull&service=      200 …plication/json  1.8k  56ms
20:06:56 HTTPS GET  …pe-west1-docker.pkg.dev /v2/artifact-registry-sync-34412/sync-01/alpine/blobs/sha256:2720e26172a023c7245fd2d59f06452cb37… 200 …n/octet-stream  1.4k 146ms
20:06:57 HTTPS GET  …pe-west1-docker.pkg.dev /v2/token?scope=repository%3Aartifact-registry-sync-34412%2Fsync-01%2Falpine%3Apull&service=      200 …plication/json  1.8k  58ms
20:06:57 HTTPS GET  …pe-west1-docker.pkg.dev /v2/artifact-registry-sync-34412/sync-01/alpine/blobs/sha256:ab6db1bc80d0a6df92d04c3fad44b944364… 302       text/html  2.9k  70ms
20:06:57 HTTPS GET  …pe-west1-docker.pkg.dev /artifacts-downloads/namespaces/artifact-registry-sync-34412/repositories/sync-01/downloads/AB38… 200 …n/octet-stream  2.7m 548ms

That 302 redirect together with the way the go default http client handles redirects makes this work. The Authentication header is added:

	// If CheckRedirect is nil, the Client uses its default policy,
	// which is to stop after 10 consecutive requests.
	CheckRedirect func(req *Request, via []*Request) error

So that means that the current implemented logic should work but I will make authentication for /artifacts-downloads more explicit (or cover potential cases where /artifacts-downloads is used in a non 302 case)

Regarding your offer. @chlins has asked in the PR if we can provide a test environment. So @chlins might be the best judge what helps the most.

[chlins]

@mrbuk Hi, markus, as we discussed in the slack, do we have any possible to have a test account?

[mrbuk]

@mrbuk Hi, markus, as we discussed in the slack, do we have any possible to have a test account?

@lindsayismith is currently looking into possibilities. When he comes online we might be able to get an update.

Is there a chance specifically for the test account conversation to continue in an email thread? If so could you DM me the folks to involve from your side via Slack?

[chlins]

Thanks for your update. Sure, I'll ping you in the slack.

[lindsayismith]

Hi, sorry for the long delay in updating.

Turns out that the URLs that were not having auth sent - the ones like /artifact-downloads etc - were already sending us an auth token in the encrypted URL. We've updated Artifact Registry so that we can accept auth from there if the headers aren't set.

So replication should work to AR now if you use the same adapter as for GCR, without having to merge any changes on the Harbor side. Would be happy to hear if this in fact working now.

[mrbuk]

Thanks @lindsayismith. Good news I have just setup a Harbor 2.6.0 installation and was able to validate the following use cases.

Pull-based replication:

1. Replicate from Artifact Registry to Harbor (p

Push-bash replication:

1. Replicate from Harbor to Artifact Registry
2. Delete from Artifact Registry when image is deleted in Harbor

The setting used was:

• Endpoint URL configured in Registry: https://europe-west1-docker.pkg.dev (using europe-west1 in this example)
• Harbor replication rule: ${GCP_PROJECT_ID}/${AR_REPOSITORY_NAME}/**

@chlins my tests have been successful. I will add this comment to the PR. So this one can actually be closed. Potentially the documentation can be adjusted.

[lindsayismith]

NIce :)

On Tue, Sep 13, 2022 at 1:47 PM Markus Bukowski @.***> wrote:

Thanks @lindsayismith https://github.com/lindsayismith. Good news I have just setup a Harbor 2.6.0 installation and was able to validate the following use cases.

Pull-based replication:

1. Replicate from Artifact Registry to Harbor (p

Push-bash replication:

1. Replicate from Harbor to Artifact Registry
2. Delete from Artifact Registry when image is deleted in Harbor

The setting used was:

• Endpoint URL configured in Registry: https://europe-west1-docker.pkg.dev (using europe-west1 in this example)
• Harbor replication rule: ${GCP_PROJECT_ID}/${AR_REPOSITORY_NAME}/**

@chlins https://github.com/chlins TL;DR my tests have been successful

— Reply to this email directly, view it on GitHub https://github.com/goharbor/harbor/issues/16973#issuecomment-1245938649, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGA2TW4N3BGQIENVAEEI7DV6DR63ANCNFSM5YKXSECQ . You are receiving this because you were mentioned.Message ID: @.***>

[chlins]

Close this issue as resolved from the AR side.

[danielcmiller]

Hi, sorry for the long delay in updating.

Turns out that the URLs that were not having auth sent - the ones like /artifact-downloads etc - were already sending us an auth token in the encrypted URL. We've updated Artifact Registry so that we can accept auth from there if the headers aren't set.

So replication should work to AR now if you use the same adapter as for GCR, without having to merge any changes on the Harbor side. Would be happy to hear if this in fact working now.

Thank you Lidsayismith! I had so many issues with gcrane NOT replicating all of the images having this direct replication to GAR is fantastic without having to upgrade our Harbor installation! Thanks again!

[lindsayismith]

Thanks for the feedback Daniel!

On Tue, Oct 11, 2022 at 1:46 PM danielcmiller @.***> wrote:

Hi, sorry for the long delay in updating.

Turns out that the URLs that were not having auth sent - the ones like /artifact-downloads etc - were already sending us an auth token in the encrypted URL. We've updated Artifact Registry so that we can accept auth from there if the headers aren't set.

So replication should work to AR now if you use the same adapter as for GCR, without having to merge any changes on the Harbor side. Would be happy to hear if this in fact working now.

Thank you Lidsayismith! I had so many issues with gcrane NOT replicating all of the images having this direct replication to GAR is fantastic without having to upgrade our Harbor installation! Thanks again!

— Reply to this email directly, view it on GitHub https://github.com/goharbor/harbor/issues/16973#issuecomment-1275257156, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGA2TSQFORK4V5Z66PQQ4DWCXG3BANCNFSM5YKXSECQ . You are receiving this because you were mentioned.Message ID: @.***>