harbor
harbor copied to clipboard
Published
20 hours ago •
goharbor
goharbor
Replication results in intermittent 500 return code
Problem description We are trying to replicate a project between a VM based harbor and a k8s based harbor. We are seeing intermittent http return code 500's for images pushed, and the replication shows these images as failed.
Both are running version 2.6.0-b035ca7c The k8s harbor has an S3 backend (NetApp) and is configured as such:
imageChartStorage:
disableredirect: true
caBundleSecretName: s3-ca-cert
type: s3
s3:
existingSecret: "s3-harbor-access-cred"
region: us-east-1
bucket: harbor
regionendpoint: https://redacted:10443
#encrypt: false
#keyid: mykeyid
secure: true
#skipverify: false
#v4auth: true
#chunksize: "5242880"
#rootdirectory: /s3/object/name/prefix
#storageclass: STANDARD
#multipartcopychunksize: "33554432"
#multipartcopymaxconcurrency: 100
#multipartcopythresholdsize: "33554432"
Logs The registry pod shows the following errors:
10.117.67.156 - - [25/Oct/2022:10:54:43 +0000] "PUT /v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/7c12a49d-661b-45da-ae8f-3c508842f369?_state=HSL04-k-y4c4zvqL3mZIj5aWHLkj0YLXKZADO12yqEh7Ik5hbWUiOiJ2bXdhcmUtdGtnL2t1YmUtdmlwLWNsb3VkLXByb3ZpZGVyIiwiVVVJRCI6IjdjMTJhNDlkLTY2MWItNDVkYS1hZThmLTNjNTA4ODQyZjM2OSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMi0xMC0yNVQxMDo1NDo0My41MjQyODY4NTZaIn0%3D&digest=sha256%3Aa694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563 HTTP/1.1" 500 218 "" "harbor-registry-client"
time="2022-10-25T10:54:52.153693146Z" level=info msg="authorized request" go.version=go1.18.5 http.request.host="edge-harbor-core:80" http.request.id=ae5aaa15-a3d9-42fc-9e83-1542cad4a70e http.request.method=POST http.request.remoteaddr=10.117.67.160 http.request.uri="/v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/" http.request.useragent=harbor-registry-client vars.name="vmware-tkg/kube-vip-cloud-provider"
time="2022-10-25T10:54:52.258025804Z" level=info msg="response completed" go.version=go1.18.5 http.request.host="edge-harbor-core:80" http.request.id=ae5aaa15-a3d9-42fc-9e83-1542cad4a70e http.request.method=POST http.request.remoteaddr=10.117.67.160 http.request.uri="/v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/" http.request.useragent=harbor-registry-client http.response.duration=221.62936ms http.response.status=202 http.response.written=0
10.117.67.156 - - [25/Oct/2022:10:54:52 +0000] "POST /v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/ HTTP/1.1" 202 0 "" "harbor-registry-client"
time="2022-10-25T10:54:52.394201813Z" level=info msg="authorized request" go.version=go1.18.5 http.request.host="edge-harbor-core:80" http.request.id=c1d47e5e-4aa7-45b8-a042-4e7380c98c7a http.request.method=PUT http.request.remoteaddr=10.117.67.160 http.request.uri="/v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/c703a1e3-92a1-4a39-ac0e-9ed0d2dbd291?_state=TAv-aZSle9uDnzgYlg2w9y_bXD3E14f2VJns66_xyUF7Ik5hbWUiOiJ2bXdhcmUtdGtnL2t1YmUtdmlwLWNsb3VkLXByb3ZpZGVyIiwiVVVJRCI6ImM3MDNhMWUzLTkyYTEtNGEzOS1hYzBlLTllZDBkMmRiZDI5MSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMi0xMC0yNVQxMDo1NDo1Mi4xNTM3NzQ2NDhaIn0%3D&digest=sha256%3Aa694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563" http.request.useragent=harbor-registry-client vars.name="vmware-tkg/kube-vip-cloud-provider" vars.uuid=c703a1e3-92a1-4a39-ac0e-9ed0d2dbd291
10.117.67.156 - - [25/Oct/2022:10:54:52 +0000] "PUT /v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/c703a1e3-92a1-4a39-ac0e-9ed0d2dbd291?_state=TAv-aZSle9uDnzgYlg2w9y_bXD3E14f2VJns66_xyUF7Ik5hbWUiOiJ2bXdhcmUtdGtnL2t1YmUtdmlwLWNsb3VkLXByb3ZpZGVyIiwiVVVJRCI6ImM3MDNhMWUzLTkyYTEtNGEzOS1hYzBlLTllZDBkMmRiZDI5MSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMi0xMC0yNVQxMDo1NDo1Mi4xNTM3NzQ2NDhaIn0%3D&digest=sha256%3Aa694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563 HTTP/1.1" 500 218 "" "harbor-registry-client"
time="2022-10-25T10:54:52.501821695Z" level=error msg="error resolving upload: s3aws: Path not found: /docker/registry/v2/repositories/vmware-tkg/kube-vip-cloud-provider/_uploads/c703a1e3-92a1-4a39-ac0e-9ed0d2dbd291/data" auth.user.name="harbor_registry_user" go.version=go1.18.5 http.request.host="edge-harbor-core:80" http.request.id=c1d47e5e-4aa7-45b8-a042-4e7380c98c7a http.request.method=PUT http.request.remoteaddr=10.117.67.160 http.request.uri="/v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/c703a1e3-92a1-4a39-ac0e-9ed0d2dbd291?_state=TAv-aZSle9uDnzgYlg2w9y_bXD3E14f2VJns66_xyUF7Ik5hbWUiOiJ2bXdhcmUtdGtnL2t1YmUtdmlwLWNsb3VkLXByb3ZpZGVyIiwiVVVJRCI6ImM3MDNhMWUzLTkyYTEtNGEzOS1hYzBlLTllZDBkMmRiZDI5MSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMi0xMC0yNVQxMDo1NDo1Mi4xNTM3NzQ2NDhaIn0%3D&digest=sha256%3Aa694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563" http.request.useragent=harbor-registry-client vars.name="vmware-tkg/kube-vip-cloud-provider" vars.uuid=c703a1e3-92a1-4a39-ac0e-9ed0d2dbd291
time="2022-10-25T10:54:52.501868873Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code=unknown err.detail="s3aws: Path not found: /docker/registry/v2/repositories/vmware-tkg/kube-vip-cloud-provider/_uploads/c703a1e3-92a1-4a39-ac0e-9ed0d2dbd291/data" err.message="unknown error" go.version=go1.18.5 http.request.host="edge-harbor-core:80" http.request.id=c1d47e5e-4aa7-45b8-a042-4e7380c98c7a http.request.method=PUT http.request.remoteaddr=10.117.67.160 http.request.uri="/v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/c703a1e3-92a1-4a39-ac0e-9ed0d2dbd291?_state=TAv-aZSle9uDnzgYlg2w9y_bXD3E14f2VJns66_xyUF7Ik5hbWUiOiJ2bXdhcmUtdGtnL2t1YmUtdmlwLWNsb3VkLXByb3ZpZGVyIiwiVVVJRCI6ImM3MDNhMWUzLTkyYTEtNGEzOS1hYzBlLTllZDBkMmRiZDI5MSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMi0xMC0yNVQxMDo1NDo1Mi4xNTM3NzQ2NDhaIn0%3D&digest=sha256%3Aa694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563" http.request.useragent=harbor-registry-client http.response.contenttype="application/json; charset=utf-8" http.response.duration=214.088293ms http.response.status=500 http.response.written=218 vars.name="vmware-tkg/kube-vip-cloud-provider" vars.uuid=c703a1e3-92a1-4a39-ac0e-9ed0d2dbd291
time="2022-10-25T10:55:08.822866276Z" level=info msg="authorized request" go.version=go1.18.5 http.request.host="edge-harbor-core:80" http.request.id=24d977af-91ee-4394-bd27-418ca926d686 http.request.method=POST http.request.remoteaddr=10.117.67.160 http.request.uri="/v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/" http.request.useragent=harbor-registry-client vars.name="vmware-tkg/kube-vip-cloud-provider"
10.117.67.156 - - [25/Oct/2022:10:55:08 +0000] "POST /v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/ HTTP/1.1" 202 0 "" "harbor-registry-client"
time="2022-10-25T10:55:08.877429729Z" level=info msg="response completed" go.version=go1.18.5 http.request.host="edge-harbor-core:80" http.request.id=24d977af-91ee-4394-bd27-418ca926d686 http.request.method=POST http.request.remoteaddr=10.117.67.160 http.request.uri="/v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/" http.request.useragent=harbor-registry-client http.response.duration=346.013165ms http.response.status=202 http.response.written=0
time="2022-10-25T10:55:09.028527143Z" level=info msg="authorized request" go.version=go1.18.5 http.request.host="edge-harbor-core:80" http.request.id=af400d6d-c597-4f1b-acaa-3693b8838e1b http.request.method=PUT http.request.remoteaddr=10.117.67.160 http.request.uri="/v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/93987a21-e5c8-4f82-a8af-8b2f34f1ebd3?_state=ySJwc1R1jt42QGS-V2bsO3hd8r2nuhYlMMZBtKi1rP17Ik5hbWUiOiJ2bXdhcmUtdGtnL2t1YmUtdmlwLWNsb3VkLXByb3ZpZGVyIiwiVVVJRCI6IjkzOTg3YTIxLWU1YzgtNGY4Mi1hOGFmLThiMmYzNGYxZWJkMyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMi0xMC0yNVQxMDo1NTowOC44MjI5MTcwM1oifQ%3D%3D&digest=sha256%3Aa694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563" http.request.useragent=harbor-registry-client vars.name="vmware-tkg/kube-vip-cloud-provider" vars.uuid=93987a21-e5c8-4f82-a8af-8b2f34f1ebd3
time="2022-10-25T10:55:09.142441533Z" level=error msg="error resolving upload: s3aws: Path not found: /docker/registry/v2/repositories/vmware-tkg/kube-vip-cloud-provider/_uploads/93987a21-e5c8-4f82-a8af-8b2f34f1ebd3/data" auth.user.name="harbor_registry_user" go.version=go1.18.5 http.request.host="edge-harbor-core:80" http.request.id=af400d6d-c597-4f1b-acaa-3693b8838e1b http.request.method=PUT http.request.remoteaddr=10.117.67.160 http.request.uri="/v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/93987a21-e5c8-4f82-a8af-8b2f34f1ebd3?_state=ySJwc1R1jt42QGS-V2bsO3hd8r2nuhYlMMZBtKi1rP17Ik5hbWUiOiJ2bXdhcmUtdGtnL2t1YmUtdmlwLWNsb3VkLXByb3ZpZGVyIiwiVVVJRCI6IjkzOTg3YTIxLWU1YzgtNGY4Mi1hOGFmLThiMmYzNGYxZWJkMyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMi0xMC0yNVQxMDo1NTowOC44MjI5MTcwM1oifQ%3D%3D&digest=sha256%3Aa694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563" http.request.useragent=harbor-registry-client vars.name="vmware-tkg/kube-vip-cloud-provider" vars.uuid=93987a21-e5c8-4f82-a8af-8b2f34f1ebd3
time="2022-10-25T10:55:09.142524708Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code=unknown err.detail="s3aws: Path not found: /docker/registry/v2/repositories/vmware-tkg/kube-vip-cloud-provider/_uploads/93987a21-e5c8-4f82-a8af-8b2f34f1ebd3/data" err.message="unknown error" go.version=go1.18.5 http.request.host="edge-harbor-core:80" http.request.id=af400d6d-c597-4f1b-acaa-3693b8838e1b http.request.method=PUT http.request.remoteaddr=10.117.67.160 http.request.uri="/v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/93987a21-e5c8-4f82-a8af-8b2f34f1ebd3?_state=ySJwc1R1jt42QGS-V2bsO3hd8r2nuhYlMMZBtKi1rP17Ik5hbWUiOiJ2bXdhcmUtdGtnL2t1YmUtdmlwLWNsb3VkLXByb3ZpZGVyIiwiVVVJRCI6IjkzOTg3YTIxLWU1YzgtNGY4Mi1hOGFmLThiMmYzNGYxZWJkMyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMi0xMC0yNVQxMDo1NTowOC44MjI5MTcwM1oifQ%3D%3D&digest=sha256%3Aa694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563" http.request.useragent=harbor-registry-client http.response.contenttype="application/json; charset=utf-8" http.response.duration=234.95163ms http.response.status=500 http.response.written=218 vars.name="vmware-tkg/kube-vip-cloud-provider" vars.uuid=93987a21-e5c8-4f82-a8af-8b2f34f1ebd3
10.117.67.156 - - [25/Oct/2022:10:55:08 +0000] "PUT /v2/vmware-tkg/kube-vip-cloud-provider/blobs/uploads/93987a21-e5c8-4f82-a8af-8b2f34f1ebd3?_state=ySJwc1R1jt42QGS-V2bsO3hd8r2nuhYlMMZBtKi1rP17Ik5hbWUiOiJ2bXdhcmUtdGtnL2t1YmUtdmlwLWNsb3VkLXByb3ZpZGVyIiwiVVVJRCI6IjkzOTg3YTIxLWU1YzgtNGY4Mi1hOGFmLThiMmYzNGYxZWJkMyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMi0xMC0yNVQxMDo1NTowOC44MjI5MTcwM1oifQ%3D%3D&digest=sha256%3Aa694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563 HTTP/1.1" 500 218 "" "harbor-registry-client"
It seems to be looking for the /data folder, which, after checking the S3 store doesn't appear to exist at all.
In the Harbor uit we see the following errors for the replication task:
2022-10-25T10:54:35Z [INFO] [/controller/replication/transfer/image/transfer.go:126]: client for source registry [type: harbor, URL: SOURCE_REDACTED, insecure: true] created
2022-10-25T10:54:35Z [INFO] [/controller/replication/transfer/image/transfer.go:136]: client for destination registry [type: harbor, URL: DESTINATION_REDACTED, insecure: true] created
2022-10-25T10:54:35Z [INFO] [/controller/replication/transfer/image/transfer.go:169]: copying vmware-tkg/kube-vip-cloud-provider:[v0.0.3](source registry) to vmware-tkg/kube-vip-cloud-provider:[v0.0.3](destination registry)...
2022-10-25T10:54:35Z [INFO] [/controller/replication/transfer/image/transfer.go:197]: copying vmware-tkg/kube-vip-cloud-provider:v0.0.3(source registry) to vmware-tkg/kube-vip-cloud-provider:v0.0.3(destination registry)...
2022-10-25T10:54:35Z [INFO] [/controller/replication/transfer/image/transfer.go:344]: pulling the manifest of artifact vmware-tkg/kube-vip-cloud-provider:v0.0.3 ...
2022-10-25T10:54:35Z [INFO] [/controller/replication/transfer/image/transfer.go:350]: the manifest of artifact vmware-tkg/kube-vip-cloud-provider:v0.0.3 pulled
2022-10-25T10:54:35Z [INFO] [/controller/replication/transfer/image/transfer.go:271]: copying the blob sha256:a694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563(the 1th running)...
2022-10-25T10:54:36Z [ERROR] [/controller/replication/transfer/image/transfer.go:333]: failed to pushing the blob sha256:a694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563, size 1047: http status code: 500, body: {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"Path":"/docker/registry/v2/repositories/vmware-tkg/kube-vip-cloud-provider/_uploads/ecc4a935-4402-45e3-8286-32bbeb56de75/data","DriverName":"s3aws"}}]}
2022-10-25T10:54:36Z [INFO] [/controller/replication/transfer/image/transfer.go:279]: will retry 2s later
2022-10-25T10:54:38Z [INFO] [/controller/replication/transfer/image/transfer.go:271]: copying the blob sha256:a694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563(the 2th running)...
2022-10-25T10:54:39Z [ERROR] [/controller/replication/transfer/image/transfer.go:333]: failed to pushing the blob sha256:a694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563, size 1047: http status code: 500, body: {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"Path":"/docker/registry/v2/repositories/vmware-tkg/kube-vip-cloud-provider/_uploads/e6501795-149c-4243-9e55-d6d1a578db76/data","DriverName":"s3aws"}}]}
2022-10-25T10:54:39Z [INFO] [/controller/replication/transfer/image/transfer.go:279]: will retry 4s later
2022-10-25T10:54:43Z [INFO] [/controller/replication/transfer/image/transfer.go:271]: copying the blob sha256:a694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563(the 3th running)...
2022-10-25T10:54:43Z [ERROR] [/controller/replication/transfer/image/transfer.go:333]: failed to pushing the blob sha256:a694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563, size 1047: http status code: 500, body: {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"Path":"/docker/registry/v2/repositories/vmware-tkg/kube-vip-cloud-provider/_uploads/7c12a49d-661b-45da-ae8f-3c508842f369/data","DriverName":"s3aws"}}]}
2022-10-25T10:54:43Z [INFO] [/controller/replication/transfer/image/transfer.go:279]: will retry 8s later
2022-10-25T10:54:51Z [INFO] [/controller/replication/transfer/image/transfer.go:271]: copying the blob sha256:a694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563(the 4th running)...
2022-10-25T10:54:52Z [ERROR] [/controller/replication/transfer/image/transfer.go:333]: failed to pushing the blob sha256:a694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563, size 1047: http status code: 500, body: {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"Path":"/docker/registry/v2/repositories/vmware-tkg/kube-vip-cloud-provider/_uploads/c703a1e3-92a1-4a39-ac0e-9ed0d2dbd291/data","DriverName":"s3aws"}}]}
2022-10-25T10:54:52Z [INFO] [/controller/replication/transfer/image/transfer.go:279]: will retry 16s later
2022-10-25T10:55:08Z [INFO] [/controller/replication/transfer/image/transfer.go:271]: copying the blob sha256:a694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563(the 5th running)...
2022-10-25T10:55:09Z [ERROR] [/controller/replication/transfer/image/transfer.go:333]: failed to pushing the blob sha256:a694e6d6d8322f6ba0e74786a7c9602231dcb993a0e8a1a839293f454f67f563, size 1047: http status code: 500, body: {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"Path":"/docker/registry/v2/repositories/vmware-tkg/kube-vip-cloud-provider/_uploads/93987a21-e5c8-4f82-a8af-8b2f34f1ebd3/data","DriverName":"s3aws"}}]}
2022-10-25T10:55:09Z [ERROR] [/controller/replication/transfer/image/transfer.go:181]: http status code: 500, body: {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"Path":"/docker/registry/v2/repositories/vmware-tkg/kube-vip-cloud-provider/_uploads/93987a21-e5c8-4f82-a8af-8b2f34f1ebd3/data","DriverName":"s3aws"}}]}
2022-10-25T10:55:09Z [ERROR] [/controller/replication/transfer/image/transfer.go:187]: got error during the whole transfer period, mark the job failure
The result looks like this in the Harbor UI:
Update: It seems it's not necessarily bound to replication. When pushing images to the project in question (vmware-tkg) we see the same errors return:
copy | exporting 10 images...
copy | will export projects.registry.vmware.com/tkg/packages/management/core-management-plugins@sha256:0f9b7d120625ecb98799d2d38fdb19c1c026c34357c2e1ba766a53871394d1d8
copy | will export projects.registry.vmware.com/tkg/packages/management/featuregates@sha256:e3c1c53aa133965dd16918702543b7bb3a2ea35f0ceed84e315e054972c48d44
copy | will export projects.registry.vmware.com/tkg/packages/management/repo@sha256:69b773add2259a90c64da6a0aeda7b2858bd161a47d3f918e299c154eb036082
copy | will export projects.registry.vmware.com/tkg/tanzu_core/featuregates/featuregates-controller-manager@sha256:3439b6bf36dda9991dd627f8689dbadf74f846233ba5692fb328e0eefe0ef149
copy | will export projects.registry.vmware.com/tkg/tanzu_core/tanzu-cli-plugins/cluster-darwin-amd64@sha256:1f17ad92027ae5a25ce4961665dcbb7722c5d413df22a871382131e7692ef171
copy | will export projects.registry.vmware.com/tkg/tanzu_core/tanzu-cli-plugins/cluster-linux-amd64@sha256:945bb212a75b9ce83cafe1defdf0fbf24dff61d5d8d688747cbf7504d948ecf1
copy | will export projects.registry.vmware.com/tkg/tanzu_core/tanzu-cli-plugins/cluster-windows-amd64@sha256:f655ec42b911dbf8459d69e2faa9ce238acf22ab979a4a2715fa12498d7523f0
copy | will export projects.registry.vmware.com/tkg/tanzu_core/tanzu-cli-plugins/kubernetes-release-darwin-amd64@sha256:d930a73c196a070c3faa280280842c7aacb85302b142b400a344f466ffe2e98e
copy | will export projects.registry.vmware.com/tkg/tanzu_core/tanzu-cli-plugins/kubernetes-release-linux-amd64@sha256:03611f00457101c6ecf56817b9cc60620e51e5346ecbba0d63305016418f1e89
copy | will export projects.registry.vmware.com/tkg/tanzu_core/tanzu-cli-plugins/kubernetes-release-windows-amd64@sha256:51edacb728027305692fd2190b5b5d908a716387567fb5b0c644d1e462703fe3
copy | exported 10 images
copy | importing 10 images...
7.86 KiB / 150.95 MiB [>---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 0.01% 21.62 KiB/s 1h59m10s
Error uploading images: PATCH https://REDACTED/v2/vmware-tkg/packages/management/repo/blobs/uploads/cd93bad7-d6f7-412f-bfcf-3be3569648f9?_state=REDACTED: UNKNOWN: unknown error; map[DriverName:s3aws Path:/docker/registry/v2/repositories/vmware-tkg/packages/management/repo/_uploads/cd93bad7-d6f7-412f-bfcf-3be3569648f9/data]
panic: send on closed channel
goroutine 194 [running]:
github.com/google/go-containerregistry/pkg/v1/remote.(*progressReader).Read(0xc0004019e0, {0xc000672000, 0xc00018b980, 0x8000})
github.com/google/[email protected]/pkg/v1/remote/write.go:326 +0x93
io.copyBuffer({0x7f8cf245ffc0, 0xc00018b980}, {0x7f8cf245ff80, 0xc0004019e0}, {0x0, 0x0, 0x0})
io/io.go:423 +0x1b2
io.Copy(...)
io/io.go:382
net/http.(*transferWriter).doBodyCopy(0xc0002aa960, {0x7f8cf245ffc0, 0xc00018b980}, {0x7f8cf245ff80, 0xc0004019e0})
net/http/transfer.go:410 +0x4d
net/http.(*transferWriter).writeBody(0xc0002aa960, {0xe9af00, 0xc000545f40})
net/http/transfer.go:357 +0x225
net/http.(*Request).write(0xc000339b00, {0xe9af00, 0xc000545f40}, 0x0, 0xc000401aa0, 0x0)
net/http/request.go:698 +0xb4e
net/http.(*persistConn).writeLoop(0xc0000ffb00)
net/http/transport.go:2389 +0x189
created by net/http.(*Transport).dialConn
net/http/transport.go:1748 +0x1e65
Download failed. Retrying....
Update: It seems it's not necessarily bound to replication. When pushing images to the project in question (vmware-tkg) we see the same errors return:
copy | exporting 10 images... copy | will export projects.registry.vmware.com/tkg/packages/management/core-management-plugins@sha256:0f9b7d120625ecb98799d2d38fdb19c1c026c34357c2e1ba766a53871394d1d8 copy | will export projects.registry.vmware.com/tkg/packages/management/featuregates@sha256:e3c1c53aa133965dd16918702543b7bb3a2ea35f0ceed84e315e054972c48d44 copy | will export projects.registry.vmware.com/tkg/packages/management/repo@sha256:69b773add2259a90c64da6a0aeda7b2858bd161a47d3f918e299c154eb036082 copy | will export projects.registry.vmware.com/tkg/tanzu_core/featuregates/featuregates-controller-manager@sha256:3439b6bf36dda9991dd627f8689dbadf74f846233ba5692fb328e0eefe0ef149 copy | will export projects.registry.vmware.com/tkg/tanzu_core/tanzu-cli-plugins/cluster-darwin-amd64@sha256:1f17ad92027ae5a25ce4961665dcbb7722c5d413df22a871382131e7692ef171 copy | will export projects.registry.vmware.com/tkg/tanzu_core/tanzu-cli-plugins/cluster-linux-amd64@sha256:945bb212a75b9ce83cafe1defdf0fbf24dff61d5d8d688747cbf7504d948ecf1 copy | will export projects.registry.vmware.com/tkg/tanzu_core/tanzu-cli-plugins/cluster-windows-amd64@sha256:f655ec42b911dbf8459d69e2faa9ce238acf22ab979a4a2715fa12498d7523f0 copy | will export projects.registry.vmware.com/tkg/tanzu_core/tanzu-cli-plugins/kubernetes-release-darwin-amd64@sha256:d930a73c196a070c3faa280280842c7aacb85302b142b400a344f466ffe2e98e copy | will export projects.registry.vmware.com/tkg/tanzu_core/tanzu-cli-plugins/kubernetes-release-linux-amd64@sha256:03611f00457101c6ecf56817b9cc60620e51e5346ecbba0d63305016418f1e89 copy | will export projects.registry.vmware.com/tkg/tanzu_core/tanzu-cli-plugins/kubernetes-release-windows-amd64@sha256:51edacb728027305692fd2190b5b5d908a716387567fb5b0c644d1e462703fe3 copy | exported 10 images copy | importing 10 images... 7.86 KiB / 150.95 MiB [>---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 0.01% 21.62 KiB/s 1h59m10s Error uploading images: PATCH https://REDACTED/v2/vmware-tkg/packages/management/repo/blobs/uploads/cd93bad7-d6f7-412f-bfcf-3be3569648f9?_state=REDACTED: UNKNOWN: unknown error; map[DriverName:s3aws Path:/docker/registry/v2/repositories/vmware-tkg/packages/management/repo/_uploads/cd93bad7-d6f7-412f-bfcf-3be3569648f9/data] panic: send on closed channel goroutine 194 [running]: github.com/google/go-containerregistry/pkg/v1/remote.(*progressReader).Read(0xc0004019e0, {0xc000672000, 0xc00018b980, 0x8000}) github.com/google/[email protected]/pkg/v1/remote/write.go:326 +0x93 io.copyBuffer({0x7f8cf245ffc0, 0xc00018b980}, {0x7f8cf245ff80, 0xc0004019e0}, {0x0, 0x0, 0x0}) io/io.go:423 +0x1b2 io.Copy(...) io/io.go:382 net/http.(*transferWriter).doBodyCopy(0xc0002aa960, {0x7f8cf245ffc0, 0xc00018b980}, {0x7f8cf245ff80, 0xc0004019e0}) net/http/transfer.go:410 +0x4d net/http.(*transferWriter).writeBody(0xc0002aa960, {0xe9af00, 0xc000545f40}) net/http/transfer.go:357 +0x225 net/http.(*Request).write(0xc000339b00, {0xe9af00, 0xc000545f40}, 0x0, 0xc000401aa0, 0x0) net/http/request.go:698 +0xb4e net/http.(*persistConn).writeLoop(0xc0000ffb00) net/http/transport.go:2389 +0x189 created by net/http.(*Transport).dialConn net/http/transport.go:1748 +0x1e65 Download failed. Retrying....
From the log it seems your network is not very well.
@dekimpew what is your backend storage for registry? Is an AWS S3, or s3-compatible storage?
This is a NetApp storegrid S3 appliance.
A very "ghost in the shell" kinda thing: The replication and upload seem to fail when the project is called vmware-tkg but it works flawlessly when it's called literally anything else.
I will be applying bucket settings I received from NetApp support today and re-evaluate.
This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.