harbor icon indicating copy to clipboard operation
harbor copied to clipboard

Published 20 hours ago • verified publisher icon goharbor

Using buildkit exporting cache reporting status: 404 Not Found

Open enajeeb opened this issue 3 years ago • 53 comments

finding : buildkit cache and image export compatibility with harbor

Work : export both output and cache layer together

https://github.com/moby/buildkit#inline-push-image-and-cache-together

buildctl --addr docker-container://buildkitd build \
  --frontend dockerfile.v0 --local context=.  --local dockerfile=docker/test --progress=plain \
  --output type=image,name=harbor-test.mysite.com/buildkit-cache/my-operator,push=true \
  --export-cache type=inline \
  --import-cache type=registry,ref=harbor-test.mysite.com/buildkit-cache/my-operator

...
#15 [base  8/16] RUN --mount=type=cache,target=/root/go/pkg/mod     --mount=...
#15 CACHED

#14 [base  7/16] COPY go.sum .
#14 CACHED

#23 [base 16/16] RUN --mount=type=cache,target=/root/go/pkg/mod     --mount=...
#23 CACHED
#24 exporting to image
#24 exporting layers done
#24 exporting manifest sha256:4cfd02e1abeb95dd038222ac9eefa36683cbd137881ac2439c02f6a5365f340c done
#24 exporting config sha256:3a39d7f051b9c2e0d392d29a205bed9607193201fe79ad38636e000cb78b510d
#24 exporting config sha256:3a39d7f051b9c2e0d392d29a205bed9607193201fe79ad38636e000cb78b510d done
#24 pushing layers
#24 pushing layers 1.1s done
#24 pushing manifest for harbor-test.mysite.com/buildkit-cache/my-operator:latest
#24 pushing manifest for harbor-test.mysite.com/buildkit-cache/my-operator:latest 0.1s done
#24 DONE 1.3s

#25 exporting cache
#25 preparing build cache for export done
#25 DONE 0.0s

Doesn't work : export output and cache layer separately

buildctl --addr docker-container://buildkitd build \
  --frontend dockerfile.v0 --local context=. --local dockerfile=docker/test --progress=plain \
  --export-cache type=registry,ref=harbor-test.mysite.com/buildkit-cache/my-operator \
  --import-cache type=registry,ref=harbor-test.mysite.com/buildkit-cache/my-operator

24 exporting cache
#24 preparing build cache for export
#24 preparing build cache for export 15.8s done
...
#24 writing config sha256:19ac525ecabdc96ffbe803eef74acf809b082c37af057873137388121f491dcd
#24 writing config sha256:19ac525ecabdc96ffbe803eef74acf809b082c37af057873137388121f491dcd 0.1s done
#24 writing manifest sha256:878798161048504bee835f947ff94fdd915232bd8b277aacd6a511e5e77bd475
#24 writing manifest sha256:878798161048504bee835f947ff94fdd915232bd8b277aacd6a511e5e77bd475 0.1s done
#24 ERROR: error writing manifest blob: failed commit on ref "sha256:878798161048504bee835f947ff94fdd915232bd8b277aacd6a511e5e77bd475": unexpected status: 404 Not Found
------
 > exporting cache:
------
error: failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:878798161048504bee835f947ff94fdd915232bd8b277aacd6a511e5e77bd475": unexpected status: 404 Not Found

Versions: Please specify the versions of following systems.

harbor version: 2.2.1 docker engine version: 20.10.5 docker-compose version 1.18.0, build 8dd22a9

enajeeb avatar Apr 28 '21 16:04 enajeeb

I am also experiencing this same error, do we have any updates on the status of this issue?

inline does not support mode=max and its something i would like to leverage

questionmorc avatar Jul 07 '21 17:07 questionmorc

Running into this issue as well, is this fixed in a recent version, or is it still in the backlog?

philraj avatar Sep 03 '21 18:09 philraj

+1 Same issue to me.

#18 ERROR: error writing manifest blob: failed commit on ref "sha256:b0b8b156437e48583181a25ad6fecc5cfa45152005d47e215e824202e6cb579c": unexpected status: 404 Not Found

Here is what I found on my gateway:

"PUT /v2/project/repo/manifests/buildcache HTTP/1.1" 404

harbor version: 2.3.2 docker engine version: 20.10.8+azure buildkit: github.com/moby/buildkit v0.9.0

longshine avatar Sep 06 '21 16:09 longshine

I'm seeing the same error using Harbor 2.1.5. Docker: 20.10.8.

arihannu avatar Sep 16 '21 13:09 arihannu

Any updates here?

tavlima avatar Sep 29 '21 00:09 tavlima

Same here...

artyom-p avatar Oct 19 '21 21:10 artyom-p

The same problem here - Harbor Version v2.3.4-7a068b12, pushing image from github actions runner (ubuntu-latest), ends with:

error: failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:xxxxxx": unexpected status: 404 Not Found

The same workflow finishes without error if switched from Harbor to Docker Hub.

This is my build-push-action configuration:


- name: Build and push
  uses: docker/build-push-action@v2
  with:
    context: argocd
    platforms: linux/amd64,linux/arm64
    push: true
    tags: ${{ env.COMPLETE_IMAGE_NAME }}:${{ env.ARGOCD_TAG }}
    cache-from: type=registry,ref=${{ env.COMPLETE_IMAGE_NAME }}:buildcache
    cache-to: type=registry,ref=${{ env.COMPLETE_IMAGE_NAME }}:buildcache,mode=max

amoravek avatar Dec 05 '21 23:12 amoravek

Hi, we would be interested by news about this as well, this is crucial for Production grade build capability with Harbor and buildkit. Alternately, has anyone used a separate generic repo exclusively for caching as a workaround? (when not using Github actions)

remidebette avatar Jan 25 '22 16:01 remidebette

This is a must have feature for production grade registries.

Xplouder avatar Jan 26 '22 00:01 Xplouder

waiting...

zhilyaev avatar Feb 01 '22 02:02 zhilyaev

core.log 
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id f7aaad8b-cd46-4065-8d29-b5074476f701 to the logger for the request HEAD /v2/library/buildcache/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/library/buildcache/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 0fd31b1b-331b-449e-93ab-fd3db69586cc to the logger for the request HEAD /v2/library/buildcache/blobs/sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/library/buildcache/blobs/sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 2428a18f-0ce6-4922-ade6-66386a75e7ba to the logger for the request HEAD /v2/library/buildcache/blobs/sha256:e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/library/buildcache/blobs/sha256:e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 38bc043d-e668-40ad-b143-3d5ccb626429 to the logger for the request HEAD /v2/library/buildcache/blobs/sha256:f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/library/buildcache/blobs/sha256:f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 6efd5d54-c117-4af0-a76f-6c39d1440d78 to the logger for the request HEAD /v2/library/buildcache/blobs/sha256:4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/library/buildcache/blobs/sha256:4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 61226881-6a37-4d74-adca-d5311e80ec9e to the logger for the request HEAD /v2/library/buildcache/manifests/latest
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/library/buildcache/manifests/latest
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"NOT_FOUND","message":"repository library/buildcache not found"}]}
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 775410c2-00ba-437c-af27-2554c5bba38f to the logger for the request PUT /v2/library/buildcache/manifests/latest
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/library/buildcache/manifests/latest
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/server/middleware/immutable/pushmf.go:51]: failed to list artifact, repository library/buildcache not found
Apr 12 23:46:34 kodi core[203889]: 2022-04-12T18:46:34Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"NOT_FOUND","message":"artifact library/buildcache@sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 not found"}]}
registry.log 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.255969557Z" level=debug msg="authorizing request" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=4188043e-cc02-4980-9e05-834b314952f7 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" http.request.useragent="buildkit/v0.10" vars.digest="sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.265055114Z" level=info msg="authorized request" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=4188043e-cc02-4980-9e05-834b314952f7 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" http.request.useragent="buildkit/v0.10" vars.digest="sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.265202669Z" level=debug msg=GetBlob auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=4188043e-cc02-4980-9e05-834b314952f7 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" http.request.useragent="buildkit/v0.10" vars.digest="sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.267510127Z" level=info msg="redis: connect redis:6379" go.version=go1.17.7 instance.id=d97298ef-7940-40ca-82c8-558e8cbf9f07 redis.connect.duration=2.097194ms service=registry version=v2.7.1.m 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.270309037Z" level=debug msg="filesystem.URLFor("/docker/registry/v2/blobs/sha256/4f/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1/data")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=4188043e-cc02-4980-9e05-834b314952f7 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" http.request.useragent="buildkit/v0.10" trace.duration=47.359µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).URLFor" trace.id=1d5ca956-9140-46c7-a321-2629b2bc61dc trace.line=217 vars.digest="sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.271962868Z" level=info msg="response completed" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=4188043e-cc02-4980-9e05-834b314952f7 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" http.request.useragent="buildkit/v0.10" http.response.contenttype="application/octet-stream" http.response.duration=18.801741ms http.response.status=200 http.response.written=0 
Apr 12 23:46:34 kodi registry[203889]: 172.20.0.9 - - [12/Apr/2022:18:46:34 +0000] "HEAD /v2/library/buildcache/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 HTTP/1.1" 200 0 "" "buildkit/v0.10"
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.300353355Z" level=debug msg="authorizing request" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=98ff7b3e-9b17-4071-8433-c70ff0266938 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139" http.request.useragent="buildkit/v0.10" vars.digest="sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.309038071Z" level=info msg="authorized request" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=98ff7b3e-9b17-4071-8433-c70ff0266938 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139" http.request.useragent="buildkit/v0.10" vars.digest="sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.309178732Z" level=debug msg=GetBlob auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=98ff7b3e-9b17-4071-8433-c70ff0266938 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139" http.request.useragent="buildkit/v0.10" vars.digest="sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: 172.20.0.9 - - [12/Apr/2022:18:46:34 +0000] "HEAD /v2/library/buildcache/blobs/sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139 HTTP/1.1" 200 0 "" "buildkit/v0.10"
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.312979703Z" level=debug msg="filesystem.URLFor("/docker/registry/v2/blobs/sha256/df/df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139/data")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=98ff7b3e-9b17-4071-8433-c70ff0266938 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139" http.request.useragent="buildkit/v0.10" trace.duration=36.075µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).URLFor" trace.id=03f6efd2-5ad8-444f-898c-2164c8105ddf trace.line=217 vars.digest="sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.314971526Z" level=info msg="response completed" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=98ff7b3e-9b17-4071-8433-c70ff0266938 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139" http.request.useragent="buildkit/v0.10" http.response.contenttype="application/octet-stream" http.response.duration=16.713527ms http.response.status=200 http.response.written=0 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.347769267Z" level=debug msg="authorizing request" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=188c3009-624c-4200-beee-77d5e48aa196 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b" http.request.useragent="buildkit/v0.10" vars.digest="sha256:e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.356591674Z" level=info msg="authorized request" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=188c3009-624c-4200-beee-77d5e48aa196 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b" http.request.useragent="buildkit/v0.10" vars.digest="sha256:e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.356732671Z" level=debug msg=GetBlob auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=188c3009-624c-4200-beee-77d5e48aa196 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b" http.request.useragent="buildkit/v0.10" vars.digest="sha256:e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.359950878Z" level=debug msg="filesystem.URLFor("/docker/registry/v2/blobs/sha256/e8/e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b/data")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=188c3009-624c-4200-beee-77d5e48aa196 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b" http.request.useragent="buildkit/v0.10" trace.duration=35.647µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).URLFor" trace.id=53676313-178f-466a-8f58-e783a5ad4e94 trace.line=217 vars.digest="sha256:e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.361483684Z" level=info msg="response completed" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=188c3009-624c-4200-beee-77d5e48aa196 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b" http.request.useragent="buildkit/v0.10" http.response.contenttype="application/octet-stream" http.response.duration=16.137327ms http.response.status=200 http.response.written=0 
Apr 12 23:46:34 kodi registry[203889]: 172.20.0.9 - - [12/Apr/2022:18:46:34 +0000] "HEAD /v2/library/buildcache/blobs/sha256:e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b HTTP/1.1" 200 0 "" "buildkit/v0.10"
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.395184813Z" level=debug msg="authorizing request" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=861e7965-1915-48ff-be67-46f871c5640e http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc" http.request.useragent="buildkit/v0.10" vars.digest="sha256:f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.404589778Z" level=info msg="authorized request" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=861e7965-1915-48ff-be67-46f871c5640e http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc" http.request.useragent="buildkit/v0.10" vars.digest="sha256:f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.404940052Z" level=debug msg=GetBlob auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=861e7965-1915-48ff-be67-46f871c5640e http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc" http.request.useragent="buildkit/v0.10" vars.digest="sha256:f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.408696395Z" level=debug msg="filesystem.URLFor("/docker/registry/v2/blobs/sha256/f5/f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc/data")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=861e7965-1915-48ff-be67-46f871c5640e http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc" http.request.useragent="buildkit/v0.10" trace.duration=37.144µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).URLFor" trace.id=ed53b742-94ac-4e27-b116-b3ed897d739a trace.line=217 vars.digest="sha256:f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.4102235Z" level=info msg="response completed" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=861e7965-1915-48ff-be67-46f871c5640e http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc" http.request.useragent="buildkit/v0.10" http.response.contenttype="application/octet-stream" http.response.duration=18.241444ms http.response.status=200 http.response.written=0 
Apr 12 23:46:34 kodi registry[203889]: 172.20.0.9 - - [12/Apr/2022:18:46:34 +0000] "HEAD /v2/library/buildcache/blobs/sha256:f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc HTTP/1.1" 200 0 "" "buildkit/v0.10"
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.446217129Z" level=debug msg="authorizing request" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=a2978381-11ab-442e-9d45-4979237668b1 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e" http.request.useragent="buildkit/v0.10" vars.digest="sha256:4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.454961343Z" level=info msg="authorized request" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=a2978381-11ab-442e-9d45-4979237668b1 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e" http.request.useragent="buildkit/v0.10" vars.digest="sha256:4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.455119488Z" level=debug msg=GetBlob auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=a2978381-11ab-442e-9d45-4979237668b1 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e" http.request.useragent="buildkit/v0.10" vars.digest="sha256:4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.458407254Z" level=debug msg="filesystem.URLFor("/docker/registry/v2/blobs/sha256/4d/4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e/data")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=a2978381-11ab-442e-9d45-4979237668b1 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e" http.request.useragent="buildkit/v0.10" trace.duration=34.852µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).URLFor" trace.id=6910511b-c10c-468c-9448-5c298de695a7 trace.line=217 vars.digest="sha256:4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e" vars.name="library/buildcache" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.459576235Z" level=info msg="response completed" go.version=go1.17.7 http.request.host=192.168.2.197 http.request.id=a2978381-11ab-442e-9d45-4979237668b1 http.request.method=HEAD http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/blobs/sha256:4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e" http.request.useragent="buildkit/v0.10" http.response.contenttype="application/octet-stream" http.response.duration=15.701449ms http.response.status=200 http.response.written=0 
Apr 12 23:46:34 kodi registry[203889]: 172.20.0.9 - - [12/Apr/2022:18:46:34 +0000] "HEAD /v2/library/buildcache/blobs/sha256:4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e HTTP/1.1" 200 0 "" "buildkit/v0.10"
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.504827736Z" level=debug msg="authorizing request" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.51233196Z" level=info msg="authorized request" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.512439072Z" level=debug msg=PutImageManifest auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.512646111Z" level=debug msg="Putting an OCI Manifest!" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.512677863Z" level=debug msg="(*manifestStore).Put" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.512700342Z" level=debug msg="(*manifestListHandler).Put" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.512723563Z" level=debug msg="(*manifestStore).Exists" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.512878161Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/library/buildcache/_manifests/revisions/sha256/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=75.124µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=19ffbfae-e616-44bf-9114-1c2dd57775aa trace.line=95 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.513035988Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/library/buildcache/_layers/sha256/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=72.56µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=fc2d3597-f4d6-4616-8632-7a479c52a7eb trace.line=95 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.51375704Z" level=debug msg="(*manifestStore).Exists" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.513931919Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/library/buildcache/_manifests/revisions/sha256/df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=67.591µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=618120d0-7acc-4bfa-90c2-91cb9081348b trace.line=95 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.514092245Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/library/buildcache/_layers/sha256/df9b9388f04ad6279a7410b85cedfdcb2208c0a003da7ab5613af71079148139/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=66.359µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=ec6aaed9-a72d-47b5-98b2-de7df668c34a trace.line=95 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.514566722Z" level=debug msg="(*manifestStore).Exists" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.514714235Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/library/buildcache/_manifests/revisions/sha256/e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=41.604µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=d6756333-8c4b-4553-8410-d577efe0bc34 trace.line=95 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.514838416Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/library/buildcache/_layers/sha256/e8fcf12219f89607bad3980b472823db314e60a450595fbedfb33e2de255998b/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=55.089µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=484ec466-28d3-40bc-8e06-597366b67c42 trace.line=95 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.515220328Z" level=debug msg="(*manifestStore).Exists" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.5153226Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/library/buildcache/_manifests/revisions/sha256/f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=40.461µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=d9d1b163-0eed-4bd0-92c4-4815ea3e8164 trace.line=95 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.515430575Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/library/buildcache/_layers/sha256/f50894d54c03423b54abdbeba757f733b69491ba4d9bcd7c0ec5b0825602b8cc/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=47.01µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=4f9bbb54-fe19-418f-adde-211715debb08 trace.line=95 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.515787752Z" level=debug msg="(*manifestStore).Exists" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.51588069Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/library/buildcache/_manifests/revisions/sha256/4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=35.874µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=c1dda54f-6ba1-4cc7-a394-6539bcb8d145 trace.line=95 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.515978181Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/library/buildcache/_layers/sha256/4ddee9f34ca9fa4a6f697f3542cfdb99d5db34a1757b927e6809e1577c97e60e/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=41.783µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=54758c7c-3061-40c7-b715-52a7bd590b4f trace.line=95 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.579459588Z" level=debug msg="filesystem.PutContent("/docker/registry/v2/repositories/library/buildcache/_manifests/revisions/sha256/fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=62.815653ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=eb38b1ac-7a37-4044-bfa9-4ce37eb901e4 trace.line=110 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.6146435Z" level=debug msg="filesystem.PutContent("/docker/registry/v2/repositories/library/buildcache/_manifests/tags/latest/index/sha256/fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=34.470421ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=b9ea4bed-f934-4aea-8e1f-c98a796ef40d trace.line=110 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.650011842Z" level=debug msg="filesystem.PutContent("/docker/registry/v2/repositories/library/buildcache/_manifests/tags/latest/current/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" trace.duration=35.223023ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=921c8598-367a-4888-afd8-104a7b287606 trace.line=110 vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.650497518Z" level=debug msg="Succeeded in putting manifest!" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" vars.name="library/buildcache" vars.reference=latest 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.650580913Z" level=info msg="response completed" go.version=go1.17.7 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=192.168.2.197 http.request.id=0d2b93b7-3049-4e7d-b081-dafe50d0718a http.request.method=PUT http.request.remoteaddr=172.20.0.1 http.request.uri="/v2/library/buildcache/manifests/latest" http.request.useragent="buildkit/v0.10" http.response.duration=147.980417ms http.response.status=201 http.response.written=0 
Apr 12 23:46:34 kodi registry[203889]: 172.20.0.9 - - [12/Apr/2022:18:46:34 +0000] "PUT /v2/library/buildcache/manifests/latest HTTP/1.1" 201 0 "" "buildkit/v0.10"
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.660240483Z" level=debug msg="authorizing request" go.version=go1.17.7 http.request.host="registry:5000" http.request.id=47a0d040-2a0d-48c6-a567-082a34b8e3ab http.request.method=GET http.request.remoteaddr="172.20.0.9:58836" http.request.uri="/v2/library/buildcache/manifests/sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" http.request.useragent=harbor-registry-client vars.name="library/buildcache" vars.reference="sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.668927024Z" level=info msg="authorized request" go.version=go1.17.7 http.request.host="registry:5000" http.request.id=47a0d040-2a0d-48c6-a567-082a34b8e3ab http.request.method=GET http.request.remoteaddr="172.20.0.9:58836" http.request.uri="/v2/library/buildcache/manifests/sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" http.request.useragent=harbor-registry-client vars.name="library/buildcache" vars.reference="sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.669073879Z" level=debug msg=GetImageManifest auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host="registry:5000" http.request.id=47a0d040-2a0d-48c6-a567-082a34b8e3ab http.request.method=GET http.request.remoteaddr="172.20.0.9:58836" http.request.uri="/v2/library/buildcache/manifests/sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" http.request.useragent=harbor-registry-client vars.name="library/buildcache" vars.reference="sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.669126828Z" level=debug msg="(*manifestStore).Get" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host="registry:5000" http.request.id=47a0d040-2a0d-48c6-a567-082a34b8e3ab http.request.method=GET http.request.remoteaddr="172.20.0.9:58836" http.request.uri="/v2/library/buildcache/manifests/sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" http.request.useragent=harbor-registry-client vars.name="library/buildcache" vars.reference="sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.669395417Z" level=debug msg="filesystem.GetContent("/docker/registry/v2/repositories/library/buildcache/_manifests/revisions/sha256/fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12/link")" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host="registry:5000" http.request.id=47a0d040-2a0d-48c6-a567-082a34b8e3ab http.request.method=GET http.request.remoteaddr="172.20.0.9:58836" http.request.uri="/v2/library/buildcache/manifests/sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" http.request.useragent=harbor-registry-client trace.duration=146.97µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=010f06ef-8c45-4330-bcb5-0aeb3b09bc88 trace.line=95 vars.name="library/buildcache" vars.reference="sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.670384674Z" level=debug msg="filesystem.Reader("/docker/registry/v2/blobs/sha256/fb/fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12/data", 0)" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host="registry:5000" http.request.id=47a0d040-2a0d-48c6-a567-082a34b8e3ab http.request.method=GET http.request.remoteaddr="172.20.0.9:58836" http.request.uri="/v2/library/buildcache/manifests/sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" http.request.useragent=harbor-registry-client trace.duration=85.65µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Reader" trace.id=d6353d2b-66a7-485f-a6a5-504352cdc2cd trace.line=125 vars.name="library/buildcache" vars.reference="sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.670597892Z" level=debug msg="(*manifestListHandler).Unmarshal" auth.user.name="harbor_registry_user" go.version=go1.17.7 http.request.host="registry:5000" http.request.id=47a0d040-2a0d-48c6-a567-082a34b8e3ab http.request.method=GET http.request.remoteaddr="172.20.0.9:58836" http.request.uri="/v2/library/buildcache/manifests/sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" http.request.useragent=harbor-registry-client vars.name="library/buildcache" vars.reference="sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" 
Apr 12 23:46:34 kodi registry[203889]: time="2022-04-12T18:46:34.671042404Z" level=info msg="response completed" go.version=go1.17.7 http.request.host="registry:5000" http.request.id=47a0d040-2a0d-48c6-a567-082a34b8e3ab http.request.method=GET http.request.remoteaddr="172.20.0.9:58836" http.request.uri="/v2/library/buildcache/manifests/sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12" http.request.useragent=harbor-registry-client http.response.contenttype="application/vnd.oci.image.index.v1+json" http.response.duration=15.745649ms http.response.status=200 http.response.written=1547 
Apr 12 23:46:34 kodi registry[203889]: 172.20.0.9 - - [12/Apr/2022:18:46:34 +0000] "GET /v2/library/buildcache/manifests/sha256:fbd2c465b8e05578b05c93f276107d819affd11fc6975b6a2851efb6fcc1bd12 HTTP/1.1" 200 1547 "" "harbor-registry-client"
command: 
docker buildx build --cache-to=type=registry,mode=max,ref=192.168.2.197/library/buildcache .

Inokenty90 avatar Apr 12 '22 18:04 Inokenty90

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] avatar Jul 06 '22 09:07 github-actions[bot]

This is still very relevant. Would there be any news about this?

remidebette avatar Jul 06 '22 13:07 remidebette

Any news on this one? It's still an issue on v2.5.

shovelend avatar Aug 25 '22 08:08 shovelend

I did some investigation around this issue, if you use --cache-to "type=registry,mode=max,ref=registry.... it will end up trying to uploaf manufest of type application/vnd.buildkit.cacheconfig.v0 which is not supported by all registry implementation, I think harbor does not support it?

I found this in logs of buildx docker instance:

time="2022-08-25T23:49:30Z" level=warning msg="reference for unknown type: application/vnd.buildkit.cacheconfig.v0"
time="2022-08-25T23:49:31Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref \"sha256:bd239a08c4b36bbc2397ea00826c1dd34c53ffbdf751685c4ae2955e631e1b81\": unexpected status: 404 Not Found\n"
time="2022-08-25T23:57:44Z" level=warning msg="reference for unknown type: application/vnd.buildkit.cacheconfig.v0"
time="2022-08-25T23:57:45Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref \"sha256:bd239a08c4b36bbc2397ea00826c1dd34c53ffbdf751685c4ae2955e631e1b81\": unexpected status: 404 Not Found\n"
time="2022-08-26T01:00:35Z" level=warning msg="reference for unknown type: application/vnd.buildkit.cacheconfig.v0"
time="2022-08-26T01:00:36Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref \"sha256:bd239a08c4b36bbc2397ea00826c1dd34c53ffbdf751685c4ae2955e631e1b81\": unexpected status: 404 Not Found\n"
time="2022-08-26T01:03:00Z" level=info msg="stopping server"
buildkitd: context canceled

can someone with knowledge confirm the support for application/vnd.buildkit.cacheconfig.v0 media type?

inline cache is probably good workaround: https://github.com/moby/buildkit#inline-push-image-and-cache-together

maciejmatuszak avatar Aug 26 '22 01:08 maciejmatuszak

Uping this one.

Any update on this? How could I help ?

XciD avatar Oct 04 '22 16:10 XciD

Uping this one.

Any update on this? How could I help ?

If any one want to fix:

diff --git a/src/controller/artifact/abstractor.go b/src/controller/artifact/abstractor.go
index 233008157..5df41b2cd 100644
--- a/src/controller/artifact/abstractor.go
+++ b/src/controller/artifact/abstractor.go
@@ -160,6 +160,15 @@ func (a *abstractor) abstractIndexMetadata(ctx context.Context, art *artifact.Ar
 	// populate the referenced artifacts
 	for _, mani := range index.Manifests {
 		digest := mani.Digest.String()
+
+		// Buildx can reference layer in manifest
+		// see: https://github.com/docker/buildx/issues/173#issuecomment-873152302
+		switch mani.MediaType {
+		case v1.MediaTypeImageLayer, v1.MediaTypeImageLayerGzip, v1.MediaTypeImageLayerNonDistributable, v1.MediaTypeImageLayerNonDistributableGzip, "application/vnd.buildkit.cacheconfig.v0":
+			art.Size += mani.Size
+			continue
+		}
+
 		// make sure the child artifact exist
 		ar, err := a.artMgr.GetByDigest(ctx, art.RepositoryName, digest)
 		if err != nil {

Then you can rebuild the images with make compile build

I don't know the side effect on garbage collection tho.

XciD avatar Oct 07 '22 09:10 XciD

Any updates on that? I can see a solution was proposed but I don't see any code changes on the official repo regarding this issue.

leoncard avatar Oct 31 '22 10:10 leoncard

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] avatar Dec 31 '22 09:12 github-actions[bot]

Not stale.

z0rc avatar Dec 31 '22 09:12 z0rc

@ywk253100 any updates on this one? +1

janekmichalik avatar Jan 05 '23 00:01 janekmichalik

still an issue for harbor 2.6.1

ushiii111798 avatar Jan 10 '23 02:01 ushiii111798

see https://github.com/goharbor/harbor/issues/18084#issuecomment-1384127627 for another reproducer & log

kiorky avatar Jan 16 '23 14:01 kiorky

Uping this one. Any update on this? How could I help ?

If any one want to fix:

diff --git a/src/controller/artifact/abstractor.go b/src/controller/artifact/abstractor.go
index 233008157..5df41b2cd 100644
--- a/src/controller/artifact/abstractor.go
+++ b/src/controller/artifact/abstractor.go
@@ -160,6 +160,15 @@ func (a *abstractor) abstractIndexMetadata(ctx context.Context, art *artifact.Ar
 	// populate the referenced artifacts
 	for _, mani := range index.Manifests {
 		digest := mani.Digest.String()
+
+		// Buildx can reference layer in manifest
+		// see: https://github.com/docker/buildx/issues/173#issuecomment-873152302
+		switch mani.MediaType {
+		case v1.MediaTypeImageLayer, v1.MediaTypeImageLayerGzip, v1.MediaTypeImageLayerNonDistributable, v1.MediaTypeImageLayerNonDistributableGzip, "application/vnd.buildkit.cacheconfig.v0":
+			art.Size += mani.Size
+			continue
+		}
+
 		// make sure the child artifact exist
 		ar, err := a.artMgr.GetByDigest(ctx, art.RepositoryName, digest)
 		if err != nil {

Then you can rebuild the images with make compile build

I don't know the side effect on garbage collection tho.

this patch works for me.

kiorky avatar Jan 16 '23 17:01 kiorky

@XciD i also took the time that artifacts are well cleaned up upon GC, and seems everything is OK.

kiorky avatar Jan 19 '23 09:01 kiorky

Old venerable bug, long awaited fix. We still have this problem and that's why our GC works weekly - not daily now.

Antiarchitect avatar Jan 19 '23 15:01 Antiarchitect

Having the same issue here. Is there any update on this?

benjamin-bergia avatar Mar 02 '23 08:03 benjamin-bergia

Having the same issue here. Is there any update on this?

+1

janekmichalik avatar Mar 02 '23 12:03 janekmichalik

same here

jone avatar Mar 10 '23 08:03 jone

The same issue in Harbor v2.6.4-77903faf

Ethonwu avatar Mar 14 '23 02:03 Ethonwu