harbor icon indicating copy to clipboard operation
harbor copied to clipboard

Published 20 hours ago • verified publisher icon goharbor

docker.io proxy cache failing after V2.2 migration

Open dioguerra opened this issue 3 years ago • 17 comments

docker pulling some images through the proxy cache does not persist them.

Expected behavior and actual behavior: Expected the images if they do not exist in proxy cache to be downloaded and persisted locally until deleted (by retain policy for example)

Steps to reproduce the problem: Setup docker.io proxy cache on harbor registry and pull through it: docker pull registry.foo.bar/docker.io/osixia/openldap:1.3.0

Versions: Installed with harbor-helm

dependencies:
- name: harbor
  version: 1.6.1
  repository: https://helm.goharbor.io

Additional context:

  • Behind OIDC setup
  • Can only log up to INFO, nothing else is shown related to this
2021-04-30T09:20:40Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:22:23Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:23:15Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:24:23Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/webhippie/redis:latest, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:27:51Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/webhippie/redis:latest, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:28:04Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:29:26Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:30:09Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:53:18Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo
2021-04-30T09:53:18Z [WARNING] [/server/middleware/repoproxy/proxy.go:139]: Artifact: docker.io/osixia/openldap:1.3.0, digest: is not found in proxy cache, fetch it from remote repo

dioguerra avatar Apr 30 '21 12:04 dioguerra

Just confirmed that this actually also affects repositories/images that worked before.

Everything that is moving through docker.io proxy is getting always re-fetched Can you please confirm @ninjadq? This is important

Previous to finding this, we where using Version 2.1 (installed from respective helm chart) and it worked flawlessly

dioguerra avatar Apr 30 '21 13:04 dioguerra

Another problem i found, testing the pull of python:2.6 from the docker.io proxy docker pull registry.foo.bar/docker.io/library/python:2.6

i get:

2021-04-30T15:33:04Z [WARNING] [/controller/proxy/local.go:132]: failed to remove artifact, error http status code: 401, body: 
2021-04-30T15:33:05Z [WARNING] [/controller/proxy/local.go:132]: failed to remove artifact, error http status code: 401, body: 
Error response from daemon: unknown: resource not found: repo docker.io/library/python, tag 2.6 not found

This is a proxy cache. The artifact indeed does not exist visually on the Portal. I can also not see it in the DB (am i searching in the correct place?):

SELECT ar.repository_name, ar.digest, ar.pull_time, ar.media_type, tag.artifact_id, tag.name
	FROM public.artifact AS ar
	INNER JOIN public.tag AS tag
		ON ar.repository_id = tag.repository_id
	WHERE repository_name LIKE 'docker.io/library/python%'
	ORDER BY tag.name ASC;

"docker.io/library/python"	"sha256:c934af72b8bd03b9804d5bde2569c320926e70392d708d113a2e71bcf98c8a20"	"2021-04-30 01:33:39.926214"	"application/vnd.docker.container.image.v1+json"	4476	"2.7"
"docker.io/library/python"	"sha256:11343a1c03c554d241be89c532850be7623753d3d4182413e58ff47432f90a03"	"2021-04-28 20:17:33.779602"	"application/vnd.docker.distribution.manifest.list.v2+json"	4476	"2.7"
"docker.io/library/python"	"sha256:f71d8ff113cd45d0c75c7bfe6793f96296eacbb11fb2e619e141fde638bda5d5"	"2021-04-30 15:20:58.198534"	"application/vnd.docker.container.image.v1+json"	4476	"2.7"
"docker.io/library/python"	"sha256:6a7fab466122ae22b3ce88d43fe6a1df1c5acc590377598bb0b79f295fee4834"	"2021-04-30 15:20:58.0927"	"application/vnd.docker.container.image.v1+json"	4476	"2.7"
"docker.io/library/python"	"sha256:7d7d82b5bbebc1e70e4cf50039ab1a5ee95cb71ea21e4c7761c47fe5bfc4c942"	"2021-04-28 20:17:36.806371"	"application/vnd.docker.distribution.manifest.list.v2+json"	4476	"2.7"
"docker.io/library/python"	"sha256:8b463666e3542432f6a0566eb90710d10b266ef56a53e2dad70ad2f9cdd07a08"	"2021-04-30 15:20:58.308285"	"application/vnd.docker.container.image.v1+json"	4476	"2.7"
"docker.io/library/python"	"sha256:21a4d2376e167740882241fc555851cd8f6d40782efcc971d42893b5e49b27d0"	"2021-04-25 01:40:19.613462"	"application/vnd.docker.distribution.manifest.list.v2+json"	4476	"2.7"
"docker.io/library/python"	"sha256:2c3f921940344546a5695d11a8118998f011de6e38d36f4edd99253330429d59"	"2021-04-30 15:20:58.376438"	"application/vnd.docker.container.image.v1+json"	4476	"2.7"
"docker.io/library/python"	"sha256:7120b6d79a61eb96dc5927204d5e8de5e8226b5e8a094e03c7ddfbad4fb82223"	"2021-04-28 18:17:29.450432"	"application/vnd.docker.distribution.manifest.list.v2+json"	4476	"2.7"
"docker.io/library/python"	"sha256:1e9e83f9721d04e7d03254809fc58b1e8a21062f577e15a9920945d81cbfae41"	"2021-04-28 20:17:35.33064"	"application/vnd.docker.distribution.manifest.list.v2+json"	4476	"2.7"
"docker.io/library/python"	"sha256:8b463666e3542432f6a0566eb90710d10b266ef56a53e2dad70ad2f9cdd07a08"	"2021-04-30 15:20:58.308285"	"application/vnd.docker.container.image.v1+json"	4509	"3.6"
"docker.io/library/python"	"sha256:21a4d2376e167740882241fc555851cd8f6d40782efcc971d42893b5e49b27d0"	"2021-04-25 01:40:19.613462"	"application/vnd.docker.distribution.manifest.list.v2+json"	4509	"3.6"
"docker.io/library/python"	"sha256:c934af72b8bd03b9804d5bde2569c320926e70392d708d113a2e71bcf98c8a20"	"2021-04-30 01:33:39.926214"	"application/vnd.docker.container.image.v1+json"	4509	"3.6"
"docker.io/library/python"	"sha256:f71d8ff113cd45d0c75c7bfe6793f96296eacbb11fb2e619e141fde638bda5d5"	"2021-04-30 15:20:58.198534"	"application/vnd.docker.container.image.v1+json"	4509	"3.6"
"docker.io/library/python"	"sha256:1e9e83f9721d04e7d03254809fc58b1e8a21062f577e15a9920945d81cbfae41"	"2021-04-28 20:17:35.33064"	"application/vnd.docker.distribution.manifest.list.v2+json"	4509	"3.6"
"docker.io/library/python"	"sha256:7120b6d79a61eb96dc5927204d5e8de5e8226b5e8a094e03c7ddfbad4fb82223"	"2021-04-28 18:17:29.450432"	"application/vnd.docker.distribution.manifest.list.v2+json"	4509	"3.6"
"docker.io/library/python"	"sha256:2c3f921940344546a5695d11a8118998f011de6e38d36f4edd99253330429d59"	"2021-04-30 15:20:58.376438"	"application/vnd.docker.container.image.v1+json"	4509	"3.6"
"docker.io/library/python"	"sha256:7d7d82b5bbebc1e70e4cf50039ab1a5ee95cb71ea21e4c7761c47fe5bfc4c942"	"2021-04-28 20:17:36.806371"	"application/vnd.docker.distribution.manifest.list.v2+json"	4509	"3.6"
"docker.io/library/python"	"sha256:6a7fab466122ae22b3ce88d43fe6a1df1c5acc590377598bb0b79f295fee4834"	"2021-04-30 15:20:58.0927"	"application/vnd.docker.container.image.v1+json"	4509	"3.6"
"docker.io/library/python"	"sha256:11343a1c03c554d241be89c532850be7623753d3d4182413e58ff47432f90a03"	"2021-04-28 20:17:33.779602"	"application/vnd.docker.distribution.manifest.list.v2+json"	4509	"3.6"
"docker.io/library/python"	"sha256:7120b6d79a61eb96dc5927204d5e8de5e8226b5e8a094e03c7ddfbad4fb82223"	"2021-04-28 18:17:29.450432"	"application/vnd.docker.distribution.manifest.list.v2+json"	4507	"3.7"
"docker.io/library/python"	"sha256:6a7fab466122ae22b3ce88d43fe6a1df1c5acc590377598bb0b79f295fee4834"	"2021-04-30 15:20:58.0927"	"application/vnd.docker.container.image.v1+json"	4507	"3.7"
"docker.io/library/python"	"sha256:c934af72b8bd03b9804d5bde2569c320926e70392d708d113a2e71bcf98c8a20"	"2021-04-30 01:33:39.926214"	"application/vnd.docker.container.image.v1+json"	4507	"3.7"
"docker.io/library/python"	"sha256:f71d8ff113cd45d0c75c7bfe6793f96296eacbb11fb2e619e141fde638bda5d5"	"2021-04-30 15:20:58.198534"	"application/vnd.docker.container.image.v1+json"	4507	"3.7"
"docker.io/library/python"	"sha256:8b463666e3542432f6a0566eb90710d10b266ef56a53e2dad70ad2f9cdd07a08"	"2021-04-30 15:20:58.308285"	"application/vnd.docker.container.image.v1+json"	4507	"3.7"
"docker.io/library/python"	"sha256:1e9e83f9721d04e7d03254809fc58b1e8a21062f577e15a9920945d81cbfae41"	"2021-04-28 20:17:35.33064"	"application/vnd.docker.distribution.manifest.list.v2+json"	4507	"3.7"
"docker.io/library/python"	"sha256:2c3f921940344546a5695d11a8118998f011de6e38d36f4edd99253330429d59"	"2021-04-30 15:20:58.376438"	"application/vnd.docker.container.image.v1+json"	4507	"3.7"
"docker.io/library/python"	"sha256:21a4d2376e167740882241fc555851cd8f6d40782efcc971d42893b5e49b27d0"	"2021-04-25 01:40:19.613462"	"application/vnd.docker.distribution.manifest.list.v2+json"	4507	"3.7"
"docker.io/library/python"	"sha256:11343a1c03c554d241be89c532850be7623753d3d4182413e58ff47432f90a03"	"2021-04-28 20:17:33.779602"	"application/vnd.docker.distribution.manifest.list.v2+json"	4507	"3.7"
"docker.io/library/python"	"sha256:7d7d82b5bbebc1e70e4cf50039ab1a5ee95cb71ea21e4c7761c47fe5bfc4c942"	"2021-04-28 20:17:36.806371"	"application/vnd.docker.distribution.manifest.list.v2+json"	4507	"3.7"
"docker.io/library/python"	"sha256:8b463666e3542432f6a0566eb90710d10b266ef56a53e2dad70ad2f9cdd07a08"	"2021-04-30 15:20:58.308285"	"application/vnd.docker.container.image.v1+json"	4514	"3.8"
"docker.io/library/python"	"sha256:11343a1c03c554d241be89c532850be7623753d3d4182413e58ff47432f90a03"	"2021-04-28 20:17:33.779602"	"application/vnd.docker.distribution.manifest.list.v2+json"	4514	"3.8"
"docker.io/library/python"	"sha256:21a4d2376e167740882241fc555851cd8f6d40782efcc971d42893b5e49b27d0"	"2021-04-25 01:40:19.613462"	"application/vnd.docker.distribution.manifest.list.v2+json"	4514	"3.8"
"docker.io/library/python"	"sha256:c934af72b8bd03b9804d5bde2569c320926e70392d708d113a2e71bcf98c8a20"	"2021-04-30 01:33:39.926214"	"application/vnd.docker.container.image.v1+json"	4514	"3.8"
"docker.io/library/python"	"sha256:7120b6d79a61eb96dc5927204d5e8de5e8226b5e8a094e03c7ddfbad4fb82223"	"2021-04-28 18:17:29.450432"	"application/vnd.docker.distribution.manifest.list.v2+json"	4514	"3.8"
"docker.io/library/python"	"sha256:7d7d82b5bbebc1e70e4cf50039ab1a5ee95cb71ea21e4c7761c47fe5bfc4c942"	"2021-04-28 20:17:36.806371"	"application/vnd.docker.distribution.manifest.list.v2+json"	4514	"3.8"
"docker.io/library/python"	"sha256:6a7fab466122ae22b3ce88d43fe6a1df1c5acc590377598bb0b79f295fee4834"	"2021-04-30 15:20:58.0927"	"application/vnd.docker.container.image.v1+json"	4514	"3.8"
"docker.io/library/python"	"sha256:2c3f921940344546a5695d11a8118998f011de6e38d36f4edd99253330429d59"	"2021-04-30 15:20:58.376438"	"application/vnd.docker.container.image.v1+json"	4514	"3.8"
"docker.io/library/python"	"sha256:f71d8ff113cd45d0c75c7bfe6793f96296eacbb11fb2e619e141fde638bda5d5"	"2021-04-30 15:20:58.198534"	"application/vnd.docker.container.image.v1+json"	4514	"3.8"
"docker.io/library/python"	"sha256:1e9e83f9721d04e7d03254809fc58b1e8a21062f577e15a9920945d81cbfae41"	"2021-04-28 20:17:35.33064"	"application/vnd.docker.distribution.manifest.list.v2+json"	4514	"3.8"
"docker.io/library/python"	"sha256:f71d8ff113cd45d0c75c7bfe6793f96296eacbb11fb2e619e141fde638bda5d5"	"2021-04-30 15:20:58.198534"	"application/vnd.docker.container.image.v1+json"	4512	"3.9"
"docker.io/library/python"	"sha256:6a7fab466122ae22b3ce88d43fe6a1df1c5acc590377598bb0b79f295fee4834"	"2021-04-30 15:20:58.0927"	"application/vnd.docker.container.image.v1+json"	4512	"3.9"
"docker.io/library/python"	"sha256:1e9e83f9721d04e7d03254809fc58b1e8a21062f577e15a9920945d81cbfae41"	"2021-04-28 20:17:35.33064"	"application/vnd.docker.distribution.manifest.list.v2+json"	4512	"3.9"
"docker.io/library/python"	"sha256:7120b6d79a61eb96dc5927204d5e8de5e8226b5e8a094e03c7ddfbad4fb82223"	"2021-04-28 18:17:29.450432"	"application/vnd.docker.distribution.manifest.list.v2+json"	4512	"3.9"
"docker.io/library/python"	"sha256:11343a1c03c554d241be89c532850be7623753d3d4182413e58ff47432f90a03"	"2021-04-28 20:17:33.779602"	"application/vnd.docker.distribution.manifest.list.v2+json"	4512	"3.9"
"docker.io/library/python"	"sha256:2c3f921940344546a5695d11a8118998f011de6e38d36f4edd99253330429d59"	"2021-04-30 15:20:58.376438"	"application/vnd.docker.container.image.v1+json"	4512	"3.9"
"docker.io/library/python"	"sha256:21a4d2376e167740882241fc555851cd8f6d40782efcc971d42893b5e49b27d0"	"2021-04-25 01:40:19.613462"	"application/vnd.docker.distribution.manifest.list.v2+json"	4512	"3.9"
"docker.io/library/python"	"sha256:7d7d82b5bbebc1e70e4cf50039ab1a5ee95cb71ea21e4c7761c47fe5bfc4c942"	"2021-04-28 20:17:36.806371"	"application/vnd.docker.distribution.manifest.list.v2+json"	4512	"3.9"
"docker.io/library/python"	"sha256:c934af72b8bd03b9804d5bde2569c320926e70392d708d113a2e71bcf98c8a20"	"2021-04-30 01:33:39.926214"	"application/vnd.docker.container.image.v1+json"	4512	"3.9"
"docker.io/library/python"	"sha256:8b463666e3542432f6a0566eb90710d10b266ef56a53e2dad70ad2f9cdd07a08"	"2021-04-30 15:20:58.308285"	"application/vnd.docker.container.image.v1+json"	4512	"3.9"

This is a proxy-cache that is public. If tag needs to be deleted it should be handled by the system. So, error returned is also misleading...

dioguerra avatar Apr 30 '21 16:04 dioguerra

@stonezdj Please have a look

ywk253100 avatar May 10 '21 04:05 ywk253100

Related:

2021-07-13T16:40:18Z [DEBUG] [/server/middleware/repoproxy/proxy.go:252]: Failed to ensure tag {Repository:docker.io/webhippie/redis Reference:latest ProjectName:docker.io Digest: Tag:latest BlobMountRepository: BlobMountProjectName: BlobMountDigest:} , error the artifact is not ready yet, failed to tag it to latest

This seems like a concurrency problem?

dioguerra avatar Jul 13 '21 17:07 dioguerra

For me looks like caching works if I use not library image, but do not work (not cached, not shown in UI) if I use library variant. harbor.domain.name/dockerhub/library/redis:6.2 <- not cached or shown in UI vs harbor.domain.name/dockerhub/bitnami/redis:5.0.13-debian-10-r5 <- looks like cached and shown in UI

vzaitsev77ionos avatar Jul 28 '21 13:07 vzaitsev77ionos

using 2.4.0 and this is still an issue...

Morriz avatar Nov 01 '21 14:11 Morriz

Same problem here on version 2.4.0 sometime images was cached, sometime never (on public or private repo)...

Steuf avatar Dec 06 '21 16:12 Steuf

Same issue on v 2.4.1

tim-sendible avatar Dec 20 '21 14:12 tim-sendible

Same issue here, using 2.4.1. For me not limited to Library images. Behaviour seems to differ when i pull it from K8s/k3s (imagepullpolicy: always) versus when pulled via manual docker pull:

K8s Pull: 2022-02-01T20:06:38Z [WARNING] [/server/middleware/repoproxy/proxy.go:143]: Artifact: gcr_cache/external-dns/external-dns:v0.10.2, digest: is not found in proxy cache, fetch it from remote repo --> nothing cached

Manual docker pull:

2022-02-01T20:08:49Z [WARNING] [/server/middleware/repoproxy/proxy.go:143]: Artifact: gcr_cache/external-dns/external-dns:v0.10.2, digest: is not found in proxy cache, fetch it from remote repo
2022-02-01T20:08:49Z [WARNING] [/server/middleware/repoproxy/proxy.go:143]: Artifact: gcr_cache/external-dns/external-dns:, digest:sha256:1fe47236861e3046b1d7b64225f60e6267d23b91a3bd24980911d4c97e641d2d is not found in proxy cache, fetch it from remote repo
2022-02-01T20:08:50Z [WARNING] [/server/middleware/repoproxy/proxy.go:143]: Artifact: gcr_cache/external-dns/external-dns:, digest:sha256:4cf7d828f7c26f62bce71fa69a53c327d1d036af3ab45d712342041b4bec3884 is not found in proxy cache, fetch it from remote repo
2022-02-01T20:09:10Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'InternalArtifact' on topic 'PUSH_ARTIFACT': ID-11, Repository-gcr_cache/external-dns/external-dns Tags-[] Digest-sha256:4cf7d828f7c26f62bce71fa69a53c327d1d036af3ab45d712342041b4bec3884 Operator-harbor#proxy-cache-service OccurAt-2022-02-01 20:09:10
2022-02-01T20:09:10Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'P2PPreheat' on topic 'PUSH_ARTIFACT': ID-11, Repository-gcr_cache/external-dns/external-dns Tags-[] Digest-sha256:4cf7d828f7c26f62bce71fa69a53c327d1d036af3ab45d712342041b4bec3884 Operator-harbor#proxy-cache-service OccurAt-2022-02-01 20:09:10
2022-02-01T20:09:10Z [INFO] [/controller/event/handler/webhook/artifact/artifact.go:75]: []
.........

--> It is cached.

For some reason some other images did get cached, i dont see the logic. In this case i use a 'Docker Registry' , configured to gcr.io. Maybe it has something to do with this.

tabnul avatar Feb 01 '22 20:02 tabnul

Same issue on v2.5.0-98e1b82f

fmalykh avatar Jun 01 '22 13:06 fmalykh

Same issue on v2.5.0-98e1b82f

sacrificeme avatar Jun 02 '22 12:06 sacrificeme

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] avatar Jul 06 '22 09:07 github-actions[bot]

This is still an issue.

tim-sendible avatar Jul 06 '22 09:07 tim-sendible

Same issue for me on v2.5.1-b0506782.

RonanQuigley avatar Aug 18 '22 11:08 RonanQuigley

Any news on this one? It's still an issue on v2.5.

shovelend avatar Aug 25 '22 08:08 shovelend

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] avatar Oct 24 '22 09:10 github-actions[bot]

still an issue in v2.6.1

msotoperdomo avatar Nov 17 '22 15:11 msotoperdomo

We also facing this issue in v2.6.1

shyamjos avatar Nov 23 '22 06:11 shyamjos

We also facing this issue in v2.6.1

Please specify the container images that fails to pull by proxy cache?

stonezdj avatar Nov 29 '22 07:11 stonezdj

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] avatar Jan 28 '23 09:01 github-actions[bot]

Having this issue too

2023-02-09T20:42:37Z [ERROR] [/server/middleware/repoproxy/proxy.go:103]: failed to proxy manifest, fallback to local, request uri: /v2/docker/itgz/minecraft-server/manifests/latest, error: http status code: 401, body:

Confirmed the creds are correct. Even remade new creds.

Same issue on three separate harbor registries.

sfxworks avatar Feb 09 '23 20:02 sfxworks

Ah it was a typo in the image. I wish the error was more clear.

sfxworks avatar Feb 09 '23 20:02 sfxworks

Summary of this issue: the image tag is not guaranteed to be cached in Harbor, because it depends on the docker client's behavior, if the tag/digest information is cached local, the docker client just sends a pull by digest request to the server, Harbor could not cache this image by tag, but cache it with digest.

stonezdj avatar Mar 06 '23 09:03 stonezdj