Tags are not replicated with event-based replication rule on a proxy-cache project

[yogeek]

Expected behavior and actual behavior:

WIth a proxy-cache project, we configured "event based" replications to mutliple ECR registries. We observed that the replicated images do not have the tags in the ECRs, only the SHA1 This causes many errors in our K8S cluster that cannot pull the tags from ECR.

But if we launch a manual replication aftet the "event based" replication, the tags are propagated.

Steps to reproduce the problem:

• configured a proxy cache project in harbor called "docker-proxy"
• configured "event based" replication to an ECR registry for this project
• docker pull $HARBOR_URL/docker-proxy/library/hello-world:latest => the image is then in harbor with latest tag
• in ECR, only the SHA1 the image is replicated ("untagged")
• launch a manual replication with the same conditions (docker-proxy/** => ECR)
• the tags are now ok

Versions:

• harbor version: [v2.2.2-56d7937f] (deployed on EKS v1.20.7-eks-d88609 with the chart version 1.6.1

Additional context:

We suspect the following root cause :

Harbor (maybe due to a change into the docker client behavior) does in fact the pull in 2 steps :

• first, he pulls the image with its SHA1
• then he adds the tag information

And the "replication to ECR" action is triggered by the pull event (not by the tag event), so the replication is done when the image only has the SHA1 but not yet the tag. Consequently, the replicated image in ECR only have the SHA1 and not the tag. Indeed, if we launch a second replication manually on this image, the tag is replicated to ECR correctly.

Here are the 2 replication logs :

• one triggered by the pull event

pull event replication

  2021-11-25T14:35:32Z [INFO] [/replication/transfer/image/transfer.go:125]: client for source registry [type: harbor, URL: http://harbor-harbor-core:80, insecure: true] created
2021-11-25T14:35:32Z [INFO] [/replication/transfer/image/transfer.go:135]: client for destination registry [type: aws-ecr, URL: https://api.ecr.eu-central-1.amazonaws.com, insecure: false] created
2021-11-25T14:35:32Z [INFO] [/replication/transfer/image/transfer.go:168]: copying cdsf/rancher/alpine-git:[sha256:c9783644fc2fae60da224564a77cb8fa8397f07cd8f4828494cc8a6326673ac7](source registry) to cdsf/rancher/alpine-git:[sha256:c9783644fc2fae60da224564a77cb8fa8397f07cd8f4828494cc8a6326673ac7](destination registry)...
2021-11-25T14:35:32Z [INFO] [/replication/transfer/image/transfer.go:192]: copying cdsf/rancher/alpine-git:sha256:c9783644fc2fae60da224564a77cb8fa8397f07cd8f4828494cc8a6326673ac7(source registry) to cdsf/rancher/alpine-git:sha256:c9783644fc2fae60da224564a77cb8fa8397f07cd8f4828494cc8a6326673ac7(destination registry)...
2021-11-25T14:35:32Z [INFO] [/replication/transfer/image/transfer.go:320]: pulling the manifest of artifact cdsf/rancher/alpine-git:sha256:c9783644fc2fae60da224564a77cb8fa8397f07cd8f4828494cc8a6326673ac7 ...
2021-11-25T14:35:33Z [INFO] [/replication/transfer/image/transfer.go:326]: the manifest of artifact cdsf/rancher/alpine-git:sha256:c9783644fc2fae60da224564a77cb8fa8397f07cd8f4828494cc8a6326673ac7 pulled
2021-11-25T14:35:33Z [INFO] [/replication/transfer/image/transfer.go:266]: copying the blob sha256:f9d0fb442560f3bad10b85cdc920ddeed03e182439aa27197994588173f74d68(the 1th running)...
2021-11-25T14:35:34Z [INFO] [/replication/transfer/image/transfer.go:268]: copy the blob sha256:f9d0fb442560f3bad10b85cdc920ddeed03e182439aa27197994588173f74d68 completed
2021-11-25T14:35:34Z [INFO] [/replication/transfer/image/transfer.go:266]: copying the blob sha256:6d987f6f42797d81a318c40d442369ba3dc124883a0964d40b0c8f4f7561d913(the 1th running)...
2021-11-25T14:35:34Z [INFO] [/replication/transfer/image/transfer.go:268]: copy the blob sha256:6d987f6f42797d81a318c40d442369ba3dc124883a0964d40b0c8f4f7561d913 completed
2021-11-25T14:35:34Z [INFO] [/replication/transfer/image/transfer.go:266]: copying the blob sha256:47af0d6632e878011700814dac8c504e967f66fe2b008dd68439d40663ca46be(the 1th running)...
2021-11-25T14:35:36Z [INFO] [/replication/transfer/image/transfer.go:268]: copy the blob sha256:47af0d6632e878011700814dac8c504e967f66fe2b008dd68439d40663ca46be completed
2021-11-25T14:35:36Z [INFO] [/replication/transfer/image/transfer.go:266]: copying the blob sha256:e1b7f492d31ae9b28cbe73d0f95fb9606682c4f532a1c14873a55bc7876bc2e5(the 1th running)...
2021-11-25T14:35:36Z [INFO] [/replication/transfer/image/transfer.go:268]: copy the blob sha256:e1b7f492d31ae9b28cbe73d0f95fb9606682c4f532a1c14873a55bc7876bc2e5 completed
2021-11-25T14:35:36Z [INFO] [/replication/transfer/image/transfer.go:349]: pushing the manifest of artifact cdsf/rancher/alpine-git:sha256:c9783644fc2fae60da224564a77cb8fa8397f07cd8f4828494cc8a6326673ac7 ...
2021-11-25T14:35:36Z [INFO] [/replication/transfer/image/transfer.go:361]: the manifest of artifact cdsf/rancher/alpine-git:sha256:c9783644fc2fae60da224564a77cb8fa8397f07cd8f4828494cc8a6326673ac7 pushed
2021-11-25T14:35:36Z [INFO] [/replication/transfer/image/transfer.go:235]: copy cdsf/rancher/alpine-git:sha256:c9783644fc2fae60da224564a77cb8fa8397f07cd8f4828494cc8a6326673ac7(source registry) to cdsf/rancher/alpine-git:sha256:c9783644fc2fae60da224564a77cb8fa8397f07cd8f4828494cc8a6326673ac7(destination registry) completed
2021-11-25T14:35:36Z [INFO] [/replication/transfer/image/transfer.go:186]: copy cdsf/rancher/alpine-git:[sha256:c9783644fc2fae60da224564a77cb8fa8397f07cd8f4828494cc8a6326673ac7](source registry) to cdsf/rancher/alpine-git:[sha256:c9783644fc2fae60da224564a77cb8fa8397f07cd8f4828494cc8a6326673ac7](destination registry) completed

• one launched manually on the same image

Manual replication

2021-11-25T14:48:03Z [INFO] [/replication/transfer/image/transfer.go:125]: client for source registry [type: harbor, URL: http://harbor-harbor-core:80, insecure: true] created
2021-11-25T14:48:03Z [INFO] [/replication/transfer/image/transfer.go:135]: client for destination registry [type: aws-ecr, URL: https://api.ecr.eu-central-1.amazonaws.com, insecure: false] created
2021-11-25T14:48:03Z [INFO] [/replication/transfer/image/transfer.go:168]: copying cdsf/rancher/alpine-git:[1.0.4](source registry) to cdsf/rancher/alpine-git:[1.0.4](destination registry)...
2021-11-25T14:48:03Z [INFO] [/replication/transfer/image/transfer.go:192]: copying cdsf/rancher/alpine-git:1.0.4(source registry) to cdsf/rancher/alpine-git:1.0.4(destination registry)...
2021-11-25T14:48:03Z [INFO] [/replication/transfer/image/transfer.go:320]: pulling the manifest of artifact cdsf/rancher/alpine-git:1.0.4 ...
2021-11-25T14:48:04Z [INFO] [/replication/transfer/image/transfer.go:326]: the manifest of artifact cdsf/rancher/alpine-git:1.0.4 pulled
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:266]: copying the blob sha256:f9d0fb442560f3bad10b85cdc920ddeed03e182439aa27197994588173f74d68(the 1th running)...
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:292]: the blob sha256:f9d0fb442560f3bad10b85cdc920ddeed03e182439aa27197994588173f74d68 already exists on the destination registry, skip
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:268]: copy the blob sha256:f9d0fb442560f3bad10b85cdc920ddeed03e182439aa27197994588173f74d68 completed
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:266]: copying the blob sha256:6d987f6f42797d81a318c40d442369ba3dc124883a0964d40b0c8f4f7561d913(the 1th running)...
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:292]: the blob sha256:6d987f6f42797d81a318c40d442369ba3dc124883a0964d40b0c8f4f7561d913 already exists on the destination registry, skip
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:268]: copy the blob sha256:6d987f6f42797d81a318c40d442369ba3dc124883a0964d40b0c8f4f7561d913 completed
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:266]: copying the blob sha256:47af0d6632e878011700814dac8c504e967f66fe2b008dd68439d40663ca46be(the 1th running)...
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:292]: the blob sha256:47af0d6632e878011700814dac8c504e967f66fe2b008dd68439d40663ca46be already exists on the destination registry, skip
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:268]: copy the blob sha256:47af0d6632e878011700814dac8c504e967f66fe2b008dd68439d40663ca46be completed
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:266]: copying the blob sha256:e1b7f492d31ae9b28cbe73d0f95fb9606682c4f532a1c14873a55bc7876bc2e5(the 1th running)...
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:292]: the blob sha256:e1b7f492d31ae9b28cbe73d0f95fb9606682c4f532a1c14873a55bc7876bc2e5 already exists on the destination registry, skip
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:268]: copy the blob sha256:e1b7f492d31ae9b28cbe73d0f95fb9606682c4f532a1c14873a55bc7876bc2e5 completed
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:349]: pushing the manifest of artifact cdsf/rancher/alpine-git:1.0.4 ...
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:361]: the manifest of artifact cdsf/rancher/alpine-git:1.0.4 pushed
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:235]: copy cdsf/rancher/alpine-git:1.0.4(source registry) to cdsf/rancher/alpine-git:1.0.4(destination registry) completed
2021-11-25T14:48:05Z [INFO] [/replication/transfer/image/transfer.go:186]: copy cdsf/rancher/alpine-git:[1.0.4](source registry) to cdsf/rancher/alpine-git:[1.0.4](destination registry) completed

Would it be possible to trigger the replication on the tag event rather than on the pull event...?

[stonezdj]

It is a limitation of proxy cache, Because the docker client pull the image in this way

HEAD  $HARBOR_URL/v2/docker-proxy/library/hello-world/tags/latest  -- Because there is no such image in local, Harbor proxy the request to the remote registry, and it returns 200 with the digest in the header.
< docker client get the digest in the HEAD request, then pulls by digest>
GET  $HARBOR_URL/v2/docker-proxy/library/hello-world@sha256:xxxxx

Because these two requests are stateless requests. Harbor only cache the request with digest. that is why there is no tag in the proxy cache, it is better to replicate these images from the remote registry to ECR.

[yogeek]

@stonezdj thanks for the details. When you say it is better to replicate these images from the remote registry to ECR: you mean we have to handle a replication between dockerhub and ECR with another tool, and not use harbor for this ? That would prevent us to benefit from harbor feature like scan, signature, retention...

But I think there is a misunderstanding : the tag is present in proxycache, but not immediately :

• 1st the sha1 is pulled in proxycache
• 2nd the replication is triggered to ECR
• 3rd the tag is set on proxycache => the tag has not been replicated
• 4th : is I launch the replication rule manually, the tag is now propagated correctly to ECR

That is why I asked if it would it be possible to implement a workaround in harbor to delay the replication once the tag is set ?

[stonezdj]

It is possible that the pull event replication is triggered before the tag is set on proxycache. then the replication replicates the artifact with digest.

[yogeek]

Indeed that's the behavior I described in my previous message. Do you think this can be fixed in harbor code please ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[yogeek]

@stonezdj any update on my previous question please ? (As I was noticed that the issue is going to be closed, I would like to know if there was any chance to improve this behavior or not before that)

[yogeek]

@stonezdj any news about the last messages please ?

[jfcavalcante]

@yogeek @stonezdj I'm facing the same issue even running the pull manually. The artifacts were being deleted by the Garbage Collection because of the delay in the Harbor to tag the image. I would like to know if there's any progress on this issue as well.

[stonezdj]

There are similar issues open https://github.com/goharbor/harbor/issues/15591, it is the behavior of docker/containerd, it cache the tag digest relationship in the local, if there is a cache in the local, it only sent the pull by digest request instead of pull by tag request. for example, run this command in client side

docker pull <harbor>/dockerhub_proxy/library/redis:latest

In server side it receives the following request, which we could found in proxy.log

un 30 14:15:37 172.30.0.1 proxy[3152]: 10.117.181.35 - "GET /api/v2.0/users/current/permissions?scope=/project/6&relative=true HTTP/1.1" 200 3387 "https://10.186.18.73/harbor/projects/6/repositories/library%2Fredis" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" 0.008 0.008 .
Jun 30 14:15:37 172.30.0.1 proxy[3152]: 10.117.181.35 - "GET /api/v2.0/projects/dockerhub_proxy/repositories/library%252Fredis HTTP/1.1" 200 182 "https://10.186.18.73/harbor/projects/6/repositories/library%2Fredis" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" 0.010 0.008 .
Jun 30 14:15:38 172.30.0.1 proxy[3152]: 10.117.181.35 - "GET /api/v2.0/projects/dockerhub_proxy/repositories/library%252Fredis/artifacts?with_tag=false&with_scan_overview=true&with_label=true&page_size=15&page=1 HTTP/1.1" 200 1489 "https://10.186.18.73/harbor/projects/6/repositories/library%2Fredis" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" 0.020 0.020 .
Jun 30 14:15:38 172.30.0.1 proxy[3152]: 10.117.181.35 - "GET /api/v2.0/labels?scope=p&project_id=6&page_size=100&page=1 HTTP/1.1" 200 3 "https://10.186.18.73/harbor/projects/6/repositories/library%2Fredis" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" 0.005 0.004 .
Jun 30 14:15:38 172.30.0.1 proxy[3152]: 10.117.181.35 - "GET /api/v2.0/labels?scope=g&page_size=100&page=1 HTTP/1.1" 200 3 "https://10.186.18.73/harbor/projects/6/repositories/library%2Fredis" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" 0.004 0.004 .
Jun 30 14:15:38 172.30.0.1 proxy[3152]: 10.117.181.35 - "GET /api/v2.0/icons/sha256:0048162a053eef4d4ce3fe7518615bef084403614f8bca43b40ae2e762e11e06 HTTP/1.1" 200 1946 "https://10.186.18.73/harbor/projects/6/repositories/library%2Fredis" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" 0.004 0.004 .
Jun 30 14:15:38 172.30.0.1 proxy[3152]: 10.117.181.35 - "GET /api/v2.0/projects/dockerhub_proxy/repositories/library%252Fredis/artifacts/sha256:31120dcdd310e9a65cbcadd504f4fe60a185bd634ab7c6a35e3e44a941904d97/tags?with_signature=true&with_immutable_status=true&page_size=8&page=1 HTTP/1.1" 200 3 "https://10.186.18.73/harbor/projects/6/repositories/library%2Fredis" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36" 0.009 0.012 .

It means that the Harbor only Get the request by digest, and Harbor is not aware its tag should be latest

[Antiarchitect]

Seems like having the same issue. In fact cache not work as cache. Having 0 artifacts after GC is finished.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

[yogeek]

/reopen