dockerhub-proxy, artifacts not being added to cache

[lknite]

From outside the kubernetes cluster I can successfully pull an image with:

docker pull harbor.<domain>/dockerhub-proxy/library/nginx

However, within the cluster I am unable to pull an image using:

k run demo --image=harbor.<domain>/dockerhub-proxy/library/nginx

Regardless of which technique is used the artifact is no longer being added to the dockerhub-proxy project. They used to be added. Not sure why they are no longer being added.

At some point the harbor-registry PVC did run out of room. I increased the size and restarted all the pods. ... if that matters.

I do see the Storage Used increase when I try to pull something new.

Please assist in how I might get artifacts to be added again as well as for pulls from within the cluster to work.

[lknite]

Not sure what logs would help to troubleshoot this, here is some logging from core immediately after trying to pull the image from within the cluster:

2022-02-17T00:03:52Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 9b6e700c04e1c782f8b41a41120288b2 to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/manifests/latest
2022-02-17T00:03:52Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/manifests/latest
2022-02-17T00:03:52Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="9b6e700c04e1c782f8b41a41120288b2"]: an unauthorized security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/manifests/latest
2022-02-17T00:03:52Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"UNAUTHORIZED","message":"authorize header needed to send HEAD to repository: authorize header needed to send HEAD to repository"}]}
2022-02-17T00:03:52Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 3e4d6a0e29f5a8d5c627b1fd0f42cb05 to the logger for the request GET /service/token
2022-02-17T00:03:52Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /service/token?scope=repository%3Adockerhub-proxy%2Flibrary%2Fnginx%3Apull&service=harbor-registry
2022-02-17T00:03:52Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="3e4d6a0e29f5a8d5c627b1fd0f42cb05"]: an unauthorized security context generated for request GET /service/token
2022-02-17T00:03:52Z [DEBUG] [/core/service/token/token.go:36]: URL for token request: /service/token?scope=repository%3Adockerhub-proxy%2Flibrary%2Fnginx%3Apull&service=harbor-registry
2022-02-17T00:03:52Z [DEBUG] [/core/service/token/creator.go:230]: scopes: [repository:dockerhub-proxy/library/nginx:pull]
2022-02-17T00:03:52Z [DEBUG] [/core/service/token/authutils.go:50]: scopes: [repository:dockerhub-proxy/library/nginx:pull]
2022/02/17 00:03:52 Model:
2022/02/17 00:03:52 g.g: _, _
2022/02/17 00:03:52 r.r: sub, obj, act
2022/02/17 00:03:52 p.p: sub, obj, act, eft
2022/02/17 00:03:52 e.e: some(where (p_eft == allow)) && !some(where (p_eft == deny))
2022/02/17 00:03:52 m.m: g(r_sub, p_sub) && keyMatch2(r_obj, p_obj) && (r_act == p_act || p_act == '*')
2022/02/17 00:03:52 Policy:
2022/02/17 00:03:52 p: sub, obj, act, eft: [[anonymous /project/2 read allow] [anonymous /project/2/label read allow] [anonymous /project/2/label list allow] [anonymous /project/2/repository list allow] [anonymous /project/2/repository pull allow] [anonymous /project/2/helm-chart read allow] [anonymous /project/2/helm-chart list allow] [anonymous /project/2/helm-chart-version read allow] [anonymous /project/2/helm-chart-version list allow] [anonymous /project/2/scan read allow] [anonymous /project/2/scanner read allow] [anonymous /project/2/tag list allow] [anonymous /project/2/artifact read allow] [anonymous /project/2/artifact list allow] [anonymous /project/2/artifact-addition read allow]]
2022/02/17 00:03:52 g: _, _: []
2022/02/17 00:03:52 Role links for: g
2022/02/17 00:03:52 
2022/02/17 00:03:52 Request: anonymous, /project/2/repository, push ---> false
2022/02/17 00:03:52 Request: anonymous, /project/2/repository, delete ---> false
2022/02/17 00:03:52 Request: anonymous, /project/2/repository, scanner-pull ---> false
2022/02/17 00:03:52 Request: anonymous, /project/2/repository, pull ---> true
2022-02-17T00:03:52Z [DEBUG] [/core/service/token/authutils.go:101]: user: , access: &{repository  dockerhub-proxy/library/nginx [pull]}
2022-02-17T00:03:52Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 8db010d6a63de6986ef83776061e2c2b to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/manifests/latest
2022-02-17T00:03:52Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/manifests/latest
2022-02-17T00:03:53Z [DEBUG] [/controller/proxy/controller.go:167]: Get the manifest list with key=cache:manifestlist:dockerhub-proxy/library/nginx:sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id d1947af6c61f5cd60146bc0fcb5b8630 to the logger for the request GET /v2/dockerhub-proxy/library/nginx/manifests/sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/manifests/sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="d1947af6c61f5cd60146bc0fcb5b8630"]: an unauthorized security context generated for request GET /v2/dockerhub-proxy/library/nginx/manifests/sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767
2022-02-17T00:03:53Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"UNAUTHORIZED","message":"authorize header needed to send HEAD to repository: authorize header needed to send HEAD to repository"}]}
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id ecd83a400adb1a944ff70cb7b795a55e to the logger for the request GET /service/token
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /service/token?scope=repository%3Adockerhub-proxy%2Flibrary%2Fnginx%3Apull&service=harbor-registry
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="ecd83a400adb1a944ff70cb7b795a55e"]: an unauthorized security context generated for request GET /service/token
2022-02-17T00:03:53Z [DEBUG] [/core/service/token/token.go:36]: URL for token request: /service/token?scope=repository%3Adockerhub-proxy%2Flibrary%2Fnginx%3Apull&service=harbor-registry
2022-02-17T00:03:53Z [DEBUG] [/core/service/token/creator.go:230]: scopes: [repository:dockerhub-proxy/library/nginx:pull]
2022-02-17T00:03:53Z [DEBUG] [/core/service/token/authutils.go:50]: scopes: [repository:dockerhub-proxy/library/nginx:pull]
2022-02-17T00:03:53Z [DEBUG] [/core/service/token/authutils.go:101]: user: , access: &{repository  dockerhub-proxy/library/nginx [pull]}
2022/02/17 00:03:53 Model:
2022/02/17 00:03:53 r.r: sub, obj, act
2022/02/17 00:03:53 p.p: sub, obj, act, eft
2022/02/17 00:03:53 e.e: some(where (p_eft == allow)) && !some(where (p_eft == deny))
2022/02/17 00:03:53 m.m: g(r_sub, p_sub) && keyMatch2(r_obj, p_obj) && (r_act == p_act || p_act == '*')
2022/02/17 00:03:53 g.g: _, _
2022/02/17 00:03:53 Policy:
2022/02/17 00:03:53 p: sub, obj, act, eft: [[anonymous /project/2 read allow] [anonymous /project/2/label read allow] [anonymous /project/2/label list allow] [anonymous /project/2/repository list allow] [anonymous /project/2/repository pull allow] [anonymous /project/2/helm-chart read allow] [anonymous /project/2/helm-chart list allow] [anonymous /project/2/helm-chart-version read allow] [anonymous /project/2/helm-chart-version list allow] [anonymous /project/2/scan read allow] [anonymous /project/2/scanner read allow] [anonymous /project/2/tag list allow] [anonymous /project/2/artifact read allow] [anonymous /project/2/artifact list allow] [anonymous /project/2/artifact-addition read allow]]
2022/02/17 00:03:53 g: _, _: []
2022/02/17 00:03:53 Role links for: g
2022/02/17 00:03:53 
2022/02/17 00:03:53 Request: anonymous, /project/2/repository, push ---> false
2022/02/17 00:03:53 Request: anonymous, /project/2/repository, delete ---> false
2022/02/17 00:03:53 Request: anonymous, /project/2/repository, scanner-pull ---> false
2022/02/17 00:03:53 Request: anonymous, /project/2/repository, pull ---> true
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id cffb6f1bbbf5cb113288d65b84e72be0 to the logger for the request GET /v2/dockerhub-proxy/library/nginx/manifests/sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/manifests/sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 81ed802f-5e1e-4451-ab8a-e5e09a5a9273 to the logger for the request GET /api/v2.0/ping
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /api/v2.0/ping
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="81ed802f-5e1e-4451-ab8a-e5e09a5a9273"]: an unauthorized security context generated for request GET /api/v2.0/ping
2022-02-17T00:03:53Z [DEBUG] [/controller/proxy/controller.go:167]: Get the manifest list with key=cache:manifestlist:dockerhub-proxy/library/nginx:sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id fba446d14e2ced58d25f05f58cd7ac07 to the logger for the request GET /v2/dockerhub-proxy/library/nginx/manifests/sha256:bb129a712c2431ecce4af8dde831e980373b26368233ef0f3b2bae9e9ec515ee
2022-02-17T00:03:53Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/manifests/sha256:bb129a712c2431ecce4af8dde831e980373b26368233ef0f3b2bae9e9ec515ee
2022-02-17T00:03:54Z [DEBUG] [/controller/proxy/controller.go:171]: Digest is not found in manifest list cache, key=cache:manifestlist:dockerhub-proxy/library/nginx:sha256:bb129a712c2431ecce4af8dde831e980373b26368233ef0f3b2bae9e9ec515ee
2022-02-17T00:03:54Z [WARNING] [/server/middleware/repoproxy/proxy.go:143]: Artifact: dockerhub-proxy/library/nginx:, digest:sha256:bb129a712c2431ecce4af8dde831e980373b26368233ef0f3b2bae9e9ec515ee is not found in proxy cache, fetch it from remote repo
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/repoproxy/proxy.go:145]: the tag is , digest is sha256:bb129a712c2431ecce4af8dde831e980373b26368233ef0f3b2bae9e9ec515ee
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 8a282c457f172a9fcad59db606b1e36e to the logger for the request GET /v2/dockerhub-proxy/library/nginx/blobs/sha256:c316d5a335a5cf324b0dc83b3da82d7608724769f6454f6d9a621f3ec2534a5a
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:c316d5a335a5cf324b0dc83b3da82d7608724769f6454f6d9a621f3ec2534a5a
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id c7eb3284-021c-4ba5-bf64-7359efb3ec67 to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:c316d5a335a5cf324b0dc83b3da82d7608724769f6454f6d9a621f3ec2534a5a
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:c316d5a335a5cf324b0dc83b3da82d7608724769f6454f6d9a621f3ec2534a5a
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="c7eb3284-021c-4ba5-bf64-7359efb3ec67"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:c316d5a335a5cf324b0dc83b3da82d7608724769f6454f6d9a621f3ec2534a5a
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 6a1e16e43ca08a7e3a6c18dd108f1790 to the logger for the request GET /v2/dockerhub-proxy/library/nginx/blobs/sha256:b559bad762bec166fd028483dd2a03f086d363ee827d8c98b7268112c508665a
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:b559bad762bec166fd028483dd2a03f086d363ee827d8c98b7268112c508665a
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 31ecc923-fb59-4b85-9013-8ecce4af95c3 to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:b559bad762bec166fd028483dd2a03f086d363ee827d8c98b7268112c508665a
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:b559bad762bec166fd028483dd2a03f086d363ee827d8c98b7268112c508665a
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="31ecc923-fb59-4b85-9013-8ecce4af95c3"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:b559bad762bec166fd028483dd2a03f086d363ee827d8c98b7268112c508665a
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id ea8311970ea80c58fc101a8536862c0e to the logger for the request GET /v2/dockerhub-proxy/library/nginx/blobs/sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 325818bbbd0ca62e51fba5b2ac7f4100 to the logger for the request GET /v2/dockerhub-proxy/library/nginx/blobs/sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id ab14382d-1704-4647-9e93-2a6daacf8c9a to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 7b76b52cb26a18770b9fc62595992a79 to the logger for the request GET /v2/dockerhub-proxy/library/nginx/blobs/sha256:78091884b7bea0fa918527207924e9993bcc21bf7f1c9687da40042ceca31ac9
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:78091884b7bea0fa918527207924e9993bcc21bf7f1c9687da40042ceca31ac9
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 3889c277e3637ff40068990f3b77b4e5 to the logger for the request GET /v2/dockerhub-proxy/library/nginx/blobs/sha256:091c283c6a66ad0edd2ab84cb10edacc00a1a7bc5277f5365c0d5c5457a75aff
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:091c283c6a66ad0edd2ab84cb10edacc00a1a7bc5277f5365c0d5c5457a75aff
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id f4498730-9ff2-4469-8e06-e288a07b46e5 to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="f4498730-9ff2-4469-8e06-e288a07b46e5"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 5876039c83c2a2001c684fa9d026fb4f to the logger for the request GET /v2/dockerhub-proxy/library/nginx/blobs/sha256:55de5851019b8f65ed6e28120c6300e35e556689d021e4b3411c7f4e90a9704b
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:55de5851019b8f65ed6e28120c6300e35e556689d021e4b3411c7f4e90a9704b
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="ab14382d-1704-4647-9e93-2a6daacf8c9a"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id d66c6a59-30b1-4693-a7e2-c42d29759dfa to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:091c283c6a66ad0edd2ab84cb10edacc00a1a7bc5277f5365c0d5c5457a75aff
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:091c283c6a66ad0edd2ab84cb10edacc00a1a7bc5277f5365c0d5c5457a75aff
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="d66c6a59-30b1-4693-a7e2-c42d29759dfa"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:091c283c6a66ad0edd2ab84cb10edacc00a1a7bc5277f5365c0d5c5457a75aff
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 2825b26d-d9cc-43a5-9c34-c87e65590d0f to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:78091884b7bea0fa918527207924e9993bcc21bf7f1c9687da40042ceca31ac9
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:78091884b7bea0fa918527207924e9993bcc21bf7f1c9687da40042ceca31ac9
2022-02-17T00:03:54Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"NOT_FOUND","message":"blob sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803 not found"}]}
2022-02-17T00:03:54Z [DEBUG] [/controller/proxy/controller.go:241]: The blob doesn't exist, proxy the request to the target server, url:library/nginx
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="2825b26d-d9cc-43a5-9c34-c87e65590d0f"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:78091884b7bea0fa918527207924e9993bcc21bf7f1c9687da40042ceca31ac9
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id a8783b75-48de-4f73-b4db-47a3aa327952 to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:55de5851019b8f65ed6e28120c6300e35e556689d021e4b3411c7f4e90a9704b
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:55de5851019b8f65ed6e28120c6300e35e556689d021e4b3411c7f4e90a9704b
2022-02-17T00:03:54Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="a8783b75-48de-4f73-b4db-47a3aa327952"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:55de5851019b8f65ed6e28120c6300e35e556689d021e4b3411c7f4e90a9704b
2022-02-17T00:03:55Z [DEBUG] [/controller/proxy/controller.go:263]: Put blob to local registry!, sourceRepo:library/nginx, localRepo:dockerhub-proxy/library/nginx, digest: sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803
2022-02-17T00:03:55Z [DEBUG] [/controller/proxy/local.go:98]: Put blob to local registry, localRepo:dockerhub-proxy/library/nginx, digest: sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803
2022-02-17T00:03:55Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id e9752e42-4a17-4372-a020-6f51dea60363 to the logger for the request POST /v2/dockerhub-proxy/library/nginx/blobs/uploads/
2022-02-17T00:03:55Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/uploads/
2022-02-17T00:03:55Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="e9752e42-4a17-4372-a020-6f51dea60363"]: a proxy cache secret security context generated for request POST /v2/dockerhub-proxy/library/nginx/blobs/uploads/
2022-02-17T00:03:55Z [DEBUG] [/server/middleware/quota/quota.go:102][action="request" middleware="quota" requestID="e9752e42-4a17-4372-a020-6f51dea60363" url="/v2/dockerhub-proxy/library/nginx/blobs/uploads/"]: no resources request for this http request, so direct to next handler
2022-02-17T00:03:55Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id a4fd96ec-389a-41ad-9327-ab03d96436ca to the logger for the request PUT /v2/dockerhub-proxy/library/nginx/blobs/uploads/8f6f40b2-4571-4d16-b90c-f9f593f9f6f2
2022-02-17T00:03:55Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/uploads/8f6f40b2-4571-4d16-b90c-f9f593f9f6f2?_state=4ZmE1WW6YyZAlT5oQd67HyceazoupxX0l549a3QyE_Z7Ik5hbWUiOiJkb2NrZXJodWItcHJveHkvbGlicmFyeS9uZ2lueCIsIlVVSUQiOiI4ZjZmNDBiMi00NTcxLTRkMTYtYjkwYy1mOWY1OTNmOWY2ZjIiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDItMTdUMDA6MDM6NTUuNDk5MzE4Mjg3WiJ9&digest=sha256%3A5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803
2022-02-17T00:03:55Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="a4fd96ec-389a-41ad-9327-ab03d96436ca"]: a proxy cache secret security context generated for request PUT /v2/dockerhub-proxy/library/nginx/blobs/uploads/8f6f40b2-4571-4d16-b90c-f9f593f9f6f2
2022-02-17T00:03:58Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 0adf9f65-d7af-4b6f-81d6-ae6a20e62c27 to the logger for the request GET /api/v2.0/ping
2022-02-17T00:03:58Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /api/v2.0/ping
2022-02-17T00:03:58Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="0adf9f65-d7af-4b6f-81d6-ae6a20e62c27"]: an unauthorized security context generated for request GET /api/v2.0/ping
2022-02-17T00:04:03Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id a06ea25d-93e5-487e-85bf-707443f4847e to the logger for the request GET /api/v2.0/ping
2022-02-17T00:04:03Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /api/v2.0/ping
2022-02-17T00:04:03Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="a06ea25d-93e5-487e-85bf-707443f4847e"]: an unauthorized security context generated for request GET /api/v2.0/ping
2022-02-17T00:04:07Z [DEBUG] [/controller/proxy/local.go:140]: checking the blob dependency: sha256:c316d5a335a5cf324b0dc83b3da82d7608724769f6454f6d9a621f3ec2534a5a
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id d0d44371-952b-4340-b3d2-85e8d99b6b6a to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:c316d5a335a5cf324b0dc83b3da82d7608724769f6454f6d9a621f3ec2534a5a
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:c316d5a335a5cf324b0dc83b3da82d7608724769f6454f6d9a621f3ec2534a5a
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="d0d44371-952b-4340-b3d2-85e8d99b6b6a"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:c316d5a335a5cf324b0dc83b3da82d7608724769f6454f6d9a621f3ec2534a5a
2022-02-17T00:04:07Z [DEBUG] [/controller/proxy/local.go:140]: checking the blob dependency: sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 6a4bb199-e95c-4013-883a-0047d8182cdc to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="6a4bb199-e95c-4013-883a-0047d8182cdc"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803
2022-02-17T00:04:07Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"NOT_FOUND","message":"blob sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803 not found"}]}
2022-02-17T00:04:07Z [DEBUG] [/controller/proxy/local.go:144]: Check dependency failed!
2022-02-17T00:04:07Z [DEBUG] [/controller/proxy/local.go:140]: checking the blob dependency: sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 4140ceba-301d-45ee-a72c-fff616a50c68 to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="4140ceba-301d-45ee-a72c-fff616a50c68"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:1ae07ab881bd848493ad54c2ba32017f94d1d8dbfd0ba41b618f17e80f834a0f
2022-02-17T00:04:07Z [DEBUG] [/controller/proxy/local.go:140]: checking the blob dependency: sha256:78091884b7bea0fa918527207924e9993bcc21bf7f1c9687da40042ceca31ac9
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 400b528c-c265-440e-938d-872173739eae to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:78091884b7bea0fa918527207924e9993bcc21bf7f1c9687da40042ceca31ac9
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:78091884b7bea0fa918527207924e9993bcc21bf7f1c9687da40042ceca31ac9
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="400b528c-c265-440e-938d-872173739eae"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:78091884b7bea0fa918527207924e9993bcc21bf7f1c9687da40042ceca31ac9
2022-02-17T00:04:07Z [DEBUG] [/controller/proxy/local.go:140]: checking the blob dependency: sha256:091c283c6a66ad0edd2ab84cb10edacc00a1a7bc5277f5365c0d5c5457a75aff
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id f6611608-6755-4d72-80b4-25083ee76575 to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:091c283c6a66ad0edd2ab84cb10edacc00a1a7bc5277f5365c0d5c5457a75aff
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:091c283c6a66ad0edd2ab84cb10edacc00a1a7bc5277f5365c0d5c5457a75aff
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="f6611608-6755-4d72-80b4-25083ee76575"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:091c283c6a66ad0edd2ab84cb10edacc00a1a7bc5277f5365c0d5c5457a75aff
2022-02-17T00:04:07Z [DEBUG] [/controller/proxy/local.go:140]: checking the blob dependency: sha256:55de5851019b8f65ed6e28120c6300e35e556689d021e4b3411c7f4e90a9704b
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 6bdd8bb7-c0bf-4c30-8fa4-0f1d98c32ae4 to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:55de5851019b8f65ed6e28120c6300e35e556689d021e4b3411c7f4e90a9704b
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:55de5851019b8f65ed6e28120c6300e35e556689d021e4b3411c7f4e90a9704b
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="6bdd8bb7-c0bf-4c30-8fa4-0f1d98c32ae4"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:55de5851019b8f65ed6e28120c6300e35e556689d021e4b3411c7f4e90a9704b
2022-02-17T00:04:07Z [DEBUG] [/controller/proxy/local.go:140]: checking the blob dependency: sha256:b559bad762bec166fd028483dd2a03f086d363ee827d8c98b7268112c508665a
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 07f4e8e5-952a-4e35-9940-2e432ac388f2 to the logger for the request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:b559bad762bec166fd028483dd2a03f086d363ee827d8c98b7268112c508665a
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/dockerhub-proxy/library/nginx/blobs/sha256:b559bad762bec166fd028483dd2a03f086d363ee827d8c98b7268112c508665a
2022-02-17T00:04:07Z [DEBUG] [/server/middleware/security/proxy_cache_secret.go:40][requestID="07f4e8e5-952a-4e35-9940-2e432ac388f2"]: a proxy cache secret security context generated for request HEAD /v2/dockerhub-proxy/library/nginx/blobs/sha256:b559bad762bec166fd028483dd2a03f086d363ee827d8c98b7268112c508665a
2022-02-17T00:04:07Z [DEBUG] [/controller/proxy/local.go:148]: Check dependency result [{application/vnd.docker.image.rootfs.diff.tar.gzip 31366257 sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803 [] map[] <nil>}]
2022-02-17T00:04:07Z [DEBUG] [/controller/proxy/manifestcache.go:175]: Current n=4 artifact: dockerhub-proxy/library/nginx:
2022-02-17T00:04:08Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id d9a50391-7e4e-4e7b-8ed8-6ed81fb2fb2d to the logger for the request GET /api/v2.0/ping
2022-02-17T00:04:08Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /api/v2.0/ping
2022-02-17T00:04:08Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="d9a50391-7e4e-4e7b-8ed8-6ed81fb2fb2d"]: an unauthorized security context generated for request GET /api/v2.0/ping

[stonezdj]

It seems that there are some blobs is skipped in local.

{"code":"NOT_FOUND","message":"blob sha256:5eb5b503b37671af16371272f9c5313a3e82f1d0756e14506704489ad9900803 not found"}]}

You could cleanup all images in your local cache and repull the image from the proxy cache.

[lknite]

I'm still working on this.

[cguedel]

Any update on this? We're seeing the same issue on a flaky connection where the registry is becoming unhealthy and then refuses to serve an image which it served a couple of minutes before just fine (so it should be cached)

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[Antiarchitect]

@lknite Any updates? We faced the same issue. It seems like a proxy cache artifact is not saved when this artifact (or sha256) already present in blob storage (from other projects maybe).

2022-07-15T17:21:12Z [ERROR] [/controller/proxy/manifestcache.go:197]: failed to push manifest, tag: , error http status code: 404, body: {"errors":[{"code":"NOT_FOUND","message":"http status code: 404, body: {\"errors\":[{\"code\":\"MANIFEST_UNKNOWN\",\"message\":\"manifest unknown\",\"detail\":{\"Name\":\"external-hub.docker.com/library/ruby\",\"Revision\":\"sha256:26cb64f809919557a7282f49eee9d7b4682033198d052aeba6011dc787e238b1\"}}]}\n"}]}

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[lknite]

I will retest, gave up on it for awhile ... will report if issue is still occurring in latest.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.