"Manifest unknown" with pull through cache

[Toudahl]

Description

The registry is unable to pull images from the remote when configured as proxy/mirror.

Reproduce

On local network server with IP 192.168.1.2:

Create config.yml with the following content, and place it in location of your choice. For me /volume3/docker/registry/config

version: 0.1
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  cache:
    blobdescriptor: inmemory
    blobdescriptorsize: 2000
  delete:
    enabled: true
proxy:
  remoteurl: https://registry-1.docker.io
  ttl: 2160h

Setup registry in docker:

docker run -d \
  -p 6000:5000 \
  --restart=always \
  --name registry \
  -v /volume3/docker/registry/data:/var/lib/registry \
  -v /volume3/docker/registry/config:/etc/distribution \
  registry:2.8.3

On windows PC: Modify %USERPROFILE%\.docker\daemon.json to contain new registry-mirrors value

{
  "registry-mirrors": [
    "http://192.168.1.2:6000"
  ]
}

Restart docker desktop. The UI will prompt for restart if value is added in UI.

Run a docker pull command, e.g. docker pull jc21/nginx-proxy-manager:2.11.3

After pull is completed, inspect data folder on server.

Expected behavior

The registry is correctly mirroring

registry version

docker image tag: 2.8.3

Additional Info

time="2024-10-03T10:54:29.523777954Z" ;level=error ;msg="response ;completed ;with ;error" ;err.code="manifest ;unknown" ;err.detail="unknown ;tag=2.11.3" ;err.message="manifest ;unknown" ;go.version=go1.20.8 ;http.request.host="192.168.1.2:6000" ;http.request.id=ec8fd1d3-0cd6-43ff-97d0-257e9896205e ;http.request.method=GET ;http.request.remoteaddr="172.17.0.1:41818" ;http.request.uri="/v2/jc21/nginx-proxy-manager/manifests/2.11.3" ;http.request.useragent="docker/27.2.0 ;go/go1.21.13 ;git-commit/3ab5c7d ;kernel/5.15.153.1-microsoft-standard-WSL2 ;os/linux ;arch/amd64 ;UpstreamClient(Docker-Client/27.2.0 ;\(windows\))" ;http.response.contenttype="application/json; ;charset=utf-8" ;http.response.duration=2.100497ms ;http.response.status=404 ;http.response.written=96 ;vars.name="jc21/nginx-proxy-manager" ;vars.reference=2.11.3 ;
172.17.0.1 ;- ;- ;[03/Oct/2024:10:54:29 ;+0000] ;"GET ;/v2/jc21/nginx-proxy-manager/manifests/2.11.3 ;HTTP/1.1" ;404 ;96 ;"" ;"docker/27.2.0 ;go/go1.21.13 ;git-commit/3ab5c7d ;kernel/5.15.153.1-microsoft-standard-WSL2 ;os/linux ;arch/amd64 ;UpstreamClient(Docker-Client/27.2.0 ;\\(windows\\))"
172.17.0.1 ;- ;- ;[03/Oct/2024:10:54:29 ;+0000] ;"HEAD ;/v2/jc21/nginx-proxy-manager/manifests/2.11.3 ;HTTP/1.1" ;404 ;96 ;"" ;"docker/27.2.0 ;go/go1.21.13 ;git-commit/3ab5c7d ;kernel/5.15.153.1-microsoft-standard-WSL2 ;os/linux ;arch/amd64 ;UpstreamClient(Docker-Client/27.2.0 ;\\(windows\\))"
time="2024-10-03T10:54:29.513423114Z" ;level=error ;msg="response ;completed;with ;error" ;err.code="manifest ;unknown" ;err.detail="unknown ;tag=2.11.3" ;err.message="manifest ;unknown" ;go.version=go1.20.8 ;http.request.host="192.168.1.2:6000" ;http.request.id=79a031df-e944-4d33-b757-964c7bf57aea ;http.request.method=HEAD ;http.request.remoteaddr="172.17.0.1:41816" ;http.request.uri="/v2/jc21/nginx-proxy-manager/manifests/2.11.3" ;http.request.useragent="docker/27.2.0 ;go/go1.21.13 ;git-commit/3ab5c7d ;kernel/5.15.153.1-microsoft-standard-WSL2 ;os/linux ;arch/amd64 ;UpstreamClient(Docker-Client/27.2.0 ;\(windows\))" ;http.response.contenttype="application/json; ;charset=utf-8" ;http.response.duration=2.288227ms ;http.response.status=404 ;http.response.written=96 ;vars.name="jc21/nginx-proxy-manager" ;vars.reference=2.11.3 ;
time="2024-10-03T10:54:29.505588511Z" ;level=info ;msg="response ;completed" ;go.version=go1.20.8 ;http.request.host="192.168.1.2:6000" ;http.request.id=c5918445-8e08-4331-af61-35644e541078 ;http.request.method=GET ;http.request.remoteaddr="172.17.0.1:41814" ;http.request.uri="/v2/" ;http.request.useragent="docker/27.2.0 ;go/go1.21.13 ;git-commit/3ab5c7d ;kernel/5.15.153.1-microsoft-standard-WSL2 ;os/linux ;arch/amd64 ;UpstreamClient(Docker-Client/27.2.0 ;\(windows\))" ;http.response.contenttype="application/json; ;charset=utf-8" ;http.response.duration=2.84768ms ;http.response.status=200 ;http.response.written=2 ;
172.17.0.1 ;- ;- ;[03/Oct/2024:10:54:29 ;+0000] ;"GET ;/v2/ ;HTTP/1.1" ;200 ;2 ;"" ;"docker/27.2.0 ;go/go1.21.13 ;git-commit/3ab5c7d ;kernel/5.15.153.1-microsoft-standard-WSL2 ;os/linux ;arch/amd64 ;UpstreamClient(Docker-Client/27.2.0 ;\\(windows\\))"

[dabaer]

I'm getting the same error in this configuration. I can pull the image manually just fine, but when run through the local registry it returns manifest unknown.

time="2025-01-12T06:09:12.462801952Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown tag=1.18.1-alpine" err.message="manifest unknown" go.version=go1.20.8 http.request.host="registry:5000" http.request.id=34f43b9f-fa4f-43ac-915e-32268c2342f6 http.request.method=GET http.request.remoteaddr="10.89.2.2:34156" http.request.uri="/v2/elixir/manifests/1.18.1-alpine" http.request.useragent="containers/5.32.2 (github.com/containers/image)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.062134223s http.response.status=404 http.response.written=103 vars.name=elixir vars.reference=1.18.1-alpine

[cowwoc]

I wasted 3 days trying to figure this out, only to realize a few minutes ago that this functionality works fine... so long as you don't interact with the proxy server directly.

Meaning, you can a proxy server and reference it from ~/.docker/daemon.json:

"registry-mirrors": [
    "http://localhost:5000"
  ]

Then, invoking docker pull nginx will actually pass the request through the proxy (returning cached images or pulling a new image from docker hub as needed).

What you can't do is invoke docker pull or docker push directly against the docker server.

docker pull will return HTTP 404 (manifest unknown) and docker push will return HTTP 500 Access denied: requested access to the resource is denied error parsing HTTP 401 response body: unexpected end of JSON input: ""

What more could you ask for, with such helpful error messages? :)

Whoever eventually picks up this issue... please please please improve these error messages. Google search results are full of people trying to figure out what the heck is going on. Too much time has been wasted trying to figure this out.

Thanks in advance and sorry for the rant 🤯

[milosgajdos]

docker pull will return HTTP 404 (manifest unknown) and docker push will return HTTP 500 Access denied: requested access to the resource is denied error parsing HTTP 401 response body: unexpected end of JSON input: ""

I agree the messages are not great for a "regular" user but they make sense:

• 404 means you are pulling a manifest (a.k.a. an image) which has not been found remotely
• 500 should probably be 403 maybe, but it's clear you're pushing something somewhere (an image to a read-only proxy) and you dont have permissions to do so

As for the error, please have a read of the errors defined by OCI spec which this project implements https://github.com/opencontainers/distribution-spec/blob/main/spec.md#error-codes

[cowwoc]

@milosgajdos I'm pretty sure there is a bug there for the 404. If you pull an image directly from the proxy, you'll get a 404 but if you configure the local Docker Engine to delegate to the proxy and pull the exact same image locally you'll get a successful response. The expectation is to get same response in both cases since ultimately the image is coming from the proxy.

For the http 500 or 403, the code is fine but the error message for humans is as useful as "something went wrong" 😀 I would love for the message to say what you wrote, that access was denied because the registry is configured as read-only proxy. For my part, I never ran across documentation or logs indicating that proxy mode causes the registry to become read-only.

Additionally, it would be helpful if the logs would summarize the configuration on startup. If the registry is configured as read-only, that would be nice to read at startup.

Thank you for all your hard work!