ProxyCache project doesn't tag saved artifacts

[Armstrong4444]

Expected behavior and actual behavior: If I pull a new image (for example: harbor.hrp.extbrain.net/quay_proxy/argoproj/argocd:v2.6.3) through ProxyCache Project I expect the image is tagged in Harbor after it is pulled and cached. Currently the image is saved without any tag.

Steps to reproduce the problem: As a container runtime I use containerd

1. Prepare Registry Endpoint in Harbor, Provider: Quay, Endpoint: https://quay.io, Access ID and Access Secret is left empty.
2. Prepare ProxyCache Project on this Registry Endpoint with name quay_proxy
3. Pull image: sudo ctr i pull --user admin:<admin_pass> <harbor_url>/quay_proxy/argoproj/argocd:v2.6.3

After the image is pulled, it is saved in the created ProxyCache Project but without tag v2.6.3 If I Use Docker for pulling the image everything is ok.

The Log message in harbor-core which is probably connected with this is:

2023-03-10T11:54:04Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:quay_proxy/argoproj/argocd Reference:v2.6.3 ProjectName:quay_proxy Digest: Tag:v2.6.3 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to v2.6.3`

Versions:

• Harbor version: 2.7.1
• containerd version: 1.6.8

I compared logs in Harbor-core in case of using Docker and in case of using Containerd. Significant log message which is logged in case of using containerd and not Docker might be:

2023-03-10T11:17:40Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="64018b0b-0684-4f01-871d-2456b0b20c8e"]: an unauthorized security context generated for request GET /v2/quay_proxy/argoproj/argocd/manifests/sha256:0fd690bd7b89bd6f947b4000de33abd53ebcd36b57216f1c675a1127707b5eef
2023-03-10T11:17:40Z [DEBUG] [/lib/http/error.go:61]: {"errors":[{"code":"UNAUTHORIZED","message":"authorize header needed to send HEAD to repository: authorize header needed to send HEAD to repository"}]}

I did some research on it and I found out that the containerd didn't set any Authorization Header in the HTTP Request.

In Postman: I sent GET request to https://<harbor_url>/v2/quay_proxy/argoproj/argocd/manifests/sha256:0fd690bd7b89bd6f947b4000de33abd53ebcd36b57216f1c675a1127707b5eef with No Authorization Header. The response is

But if I made up any Basic Authorization Header with any credentials in it I got a correct response:

I also tried to send a request to quay.io directly and without any Authorization Header and the response was correct too unlike Harbor.

Code snippet which might be relevant to this (?) https://github.com/goharbor/harbor/blob/a4c577f9ec4f18396207a5e686433a6ba203d4ef/src/server/middleware/v2auth/auth.go#L65-L78

I am not sure if this the cause of the proxy cache tagging issue of course.

I can give any additional information or log messages if needed. Thank you.

[Armstrong4444]

I add that if the provider of the Registry Endpoint is set to DockerHub, everything seems to be ok. But not otherwise.

[dfang]

I can pull image through proxy cache, but i can't see any items in proxy repository( Proxy Cached to Docker hub)

[root@localhost ~]# docker pull 192.168.1.102:8008/proxy/consul:1.15
1.15: Pulling from proxy/consul
Digest: sha256:28b50d5bfc7f95d515d19590d44f1cf566e550545e653452c2046c92b6a07e7e
Status: Image is up to date for 192.168.1.102:8008/proxy/consul:1.15
192.168.1.102:8008/proxy/consul:1.15

harbor is installed by bitnami/harbor-core
and  Version is v2.7.1-6015b3e from About

[stonezdj]

I couldn't reproduce this issue with my env and the same image. quay.io/argoproj/argocd:v2.6.3

Because harbor proxy rely on HEAD request to link the digest and the tag.

If the tag and its digest information is cached locally, then it won't send the HEAD request to the server, then the Harbor doesn't link the tag and the digest.

[lusu007]

I think I got a similar issue.

When pulling e.g. library/node it gets listed and tagged but when I try pulling e.g. alpine/helm I got the following error message:

2023-05-31T16:33:52Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.0 ProjectName:proxy Digest: Tag:3.12.0 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.0
2023-05-31T16:33:53Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.0 ProjectName:proxy Digest: Tag:3.12.0 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.0

The pull works as expected but the tagging goes wrong and its not shown in the UI.

[lusu007]

At the moment Harbor spams log messages and the Artifact doesn't show up in the UI.

The weird thing is that alpine/helm:latest shows up in the UI and is tagged.

2023-06-21T20:21:22Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="ef0cadd4-a16b-4a35-9666-110f5982e4cc"]: an unauthorized security context generated for request GET /api/v2.0/ping
2023-06-21T20:21:22Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 437dc2ca-310e-4415-bade-3d4c27ebdd92 to the logger for the request GET /api/v2.0/ping
2023-06-21T20:21:22Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /api/v2.0/ping
2023-06-21T20:21:22Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="437dc2ca-310e-4415-bade-3d4c27ebdd92"]: an unauthorized security context generated for request GET /api/v2.0/ping
2023-06-21T20:21:22Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:23Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:24Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:24Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:26Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:26Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:27Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:28Z [DEBUG] [/pkg/task/dao/execution.go:457]: skip to refresh, no outdate execution status found
2023-06-21T20:21:29Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:29Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:29Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:29Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:29Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:30Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:30Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:30Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:31Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:31Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:31Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:32Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 73ada95d-a4c1-4e66-8f72-de17273d90bb to the logger for the request GET /api/v2.0/ping
2023-06-21T20:21:32Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /api/v2.0/ping
2023-06-21T20:21:32Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="73ada95d-a4c1-4e66-8f72-de17273d90bb"]: an unauthorized security context generated for request GET /api/v2.0/ping
2023-06-21T20:21:32Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 62ef76a2-6b49-431d-a898-0c015a2a71f5 to the logger for the request GET /api/v2.0/ping
2023-06-21T20:21:32Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /api/v2.0/ping
2023-06-21T20:21:32Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="62ef76a2-6b49-431d-a898-0c015a2a71f5"]: an unauthorized security context generated for request GET /api/v2.0/ping
2023-06-21T20:21:32Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:33Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:36Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:36Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:37Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:39Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:39Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:39Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:39Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:39Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:40Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:40Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:40Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:41Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:41Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:42Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id eb2b59aa-7443-41e4-9052-cd7a90330e8f to the logger for the request GET /api/v2.0/ping
2023-06-21T20:21:42Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /api/v2.0/ping
2023-06-21T20:21:42Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="eb2b59aa-7443-41e4-9052-cd7a90330e8f"]: an unauthorized security context generated for request GET /api/v2.0/ping
2023-06-21T20:21:42Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 82ec9e81-674d-4844-a579-28de3264e94b to the logger for the request GET /api/v2.0/ping
2023-06-21T20:21:42Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /api/v2.0/ping
2023-06-21T20:21:42Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="82ec9e81-674d-4844-a579-28de3264e94b"]: an unauthorized security context generated for request GET /api/v2.0/ping
2023-06-21T20:21:42Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1
2023-06-21T20:21:43Z [DEBUG] [/server/middleware/repoproxy/proxy.go:308]: Failed to ensure tag {Repository:proxy/alpine/helm Reference:3.12.1 ProjectName:proxy Digest: Tag:3.12.1 BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to 3.12.1

alpine/helm isn't the only image that cant be tagged. The following images are also affected by this:

• node
• cytopia/yamllint
• mvdan/shfmt
• koalaman/shellcheck-alpine

I'm happy to provide more information if needed.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[meyfa]

This is still relevant.

[isugimpy]

Seeing this issue on 2.8.4 and 2.9.0. Did not see it on 2.5.1.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[lusu007]

This issue still exists.

[isugimpy]

@stonezdj Any further thoughts on this issue? As far as I can tell this is actually problematic, because it means that when a client reaches out to Harbor to pull an image by tag, the pull can fail if the backing host that the proxy cache is fronting is unavailable.

[cnmcavoy]

I've noticed this as well affecting our proxy cache setup. One potentially useful detail is that this seems to only happen for pulling multi-arch containers through the proxy cache. I suspect this is because they use a manifest list, rather than a v2 manifest.

Pulling every possible architecture specified in such a manifest list does seem to cause harbor to tag the artifacts correctly.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[meyfa]

Since there have been multiple reproductions of this issue from different people with different setups, I'm pretty sure this can be considered a confirmed bug.

Can it be marked https://github.com/goharbor/harbor/labels/never-stale, please? Even if fixing it takes time, the bug likely won't go away on its own.

[n9]

Unstale.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[lusu007]

Can it be marked never-stale , please?

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.