docker-registry-ui Cannot push to HTTPS registry

Hey @Joxit :)

I've deployed the helm chart in a simple single node minikube setup and generated certificates using cert-manager.

Had to mount your nginx configuration manually according to this example since it isn't supported in the helm-chart.

After a while, I've managed to deploy everything successfully and access the registry via HTTPS (without any errors)

Here's my nginx config:

  nginx.conf: |
    server {
      listen              443 ssl;
      ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
      ssl_ciphers         HIGH:!aNULL:!MD5;
      ssl_certificate     /etc/nginx/certs/tls.crt;
      ssl_certificate_key /etc/nginx/certs/tls.key;
      root /usr/share/nginx/html;

      # disable any limits to avoid HTTP 413 for large image uploads
      client_max_body_size 0;

      location /v2 {
          # Do not allow connections from docker 1.5 and earlier
          # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
          if ($http_user_agent ~ "^(docker\/1\.(3|4|5(.[0-9]-dev))|Go ).*$" ) {
              return 404;
          }
          proxy_pass http://sternum-registry-registry-server:5000;
      }
    }

    server {
      listen 80;
      location /  {
        # Force HTTPS
        return 301 https://$host:4443$request_uri;
      }
    }

The issue is only when I try to push images from the machine's terminal (amazon-linux-2023) to the registry. Since I use self-signed cert, I believe I need to make it trusted by the OS and the docker engine. Tried everything that the web has to offer but still the last stage I managed to reach is that docker showing me this:

5f70bf18a086: Retrying in 3 seconds 
584da9207b40: Retrying in 3 seconds 
faed71b7cc91: Retrying in 3 seconds 
ec6d18f3735b: Retrying in 3 seconds 
fcb3e6ea3f31: Retrying in 3 seconds 
63ca1fbb43ae: Waiting

and after the timeout:

tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "*.us-west-2.compute.amazonaws.com")

Registry-UI log:

xx.xxx.xx.xx - - [07/Apr/2025:10:12:07 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \x5C(darwin\x5C))" "-"
xx.xxx.xx.xx - - [07/Apr/2025:10:12:07 +0000] "HEAD /v2/timberio/vector/blobs/sha256:25429511a353a171fca8e5f090186840c3c19c59effc3bd505f54754d17ce86b HTTP/1.1" 404 0 "-" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \x5C(darwin\x5C))" "-"
xx.xxx.xx.xx - - [07/Apr/2025:10:12:07 +0000] "HEAD /v2/timberio/vector/blobs/sha256:c9b718423d7dfa8b4ac4f5d33e853621c044e9973942b98d799f70b4f418f71a HTTP/1.1" 404 0 "-" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \x5C(darwin\x5C))" "-"
xx.xxx.xx.xx - - [07/Apr/2025:10:12:07 +0000] "HEAD /v2/timberio/vector/blobs/sha256:dc8812030f8f6f0541690d058efbb028f4a0c67a1b2dc012e6745bd616bad930 HTTP/1.1" 404 0 "-" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \x5C(darwin\x5C))" "-"
xx.xxx.xx.xx - - [07/Apr/2025:10:12:07 +0000] "HEAD /v2/timberio/vector/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 HTTP/1.1" 404 0 "-" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \x5C(darwin\x5C))" "-"
xx.xxx.xx.xx - - [07/Apr/2025:10:12:07 +0000] "HEAD /v2/timberio/vector/blobs/sha256:f771120c9c9899a957f14c0e7b131c26168deb7a9e5f7db2d365c27d42b7a8c3 HTTP/1.1" 404 0 "-" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \x5C(darwin\x5C))" "-"
xx.xxx.xx.xx - - [07/Apr/2025:10:12:08 +0000] "POST /v2/timberio/vector/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \x5C(darwin\x5C))" "-"
xx.xxx.xx.xx - - [07/Apr/2025:10:12:08 +0000] "POST /v2/timberio/vector/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \x5C(darwin\x5C))" "-"
xx.xxx.xx.xx - - [07/Apr/2025:10:12:08 +0000] "POST /v2/timberio/vector/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \x5C(darwin\x5C))" "-"
xx.xxx.xx.xx - - [07/Apr/2025:10:12:08 +0000] "POST /v2/timberio/vector/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \x5C(darwin\x5C))" "-"
xx.xxx.xx.xx - - [07/Apr/2025:10:12:08 +0000] "POST /v2/timberio/vector/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \x5C(darwin\x5C))" "-"

Registry-Server log:

yy.yyy.yyy.yyy - - [07/Apr/2025:10:12:07 +0000] "GET /v2/ HTTP/1.0" 200 2 "" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \\(darwin\\))"
time="2025-04-07T10:12:07.947309789Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:25429511a353a171fca8e5f090186840c3c19c59effc3bd505f54754d17ce86b err.message="blob unknown to registry" go.version=go1.20.8 http.request.host="sternum-registry-registry-server:5000" http.request.id=53ce5103-35b1-4caf-b509-0e2fbe7ffd6c http.request.method=HEAD http.request.remoteaddr="yy.yyy.yyy.yyy:49084" http.request.uri="/v2/timberio/vector/blobs/sha256:25429511a353a171fca8e5f090186840c3c19c59effc3bd505f54754d17ce86b" http.request.useragent="docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.903291ms http.response.status=404 http.response.written=157 vars.digest="sha256:25429511a353a171fca8e5f090186840c3c19c59effc3bd505f54754d17ce86b" vars.name="timberio/vector"
yy.yyy.yyy.yyy - - [07/Apr/2025:10:12:07 +0000] "HEAD /v2/timberio/vector/blobs/sha256:25429511a353a171fca8e5f090186840c3c19c59effc3bd505f54754d17ce86b HTTP/1.0" 404 157 "" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \\(darwin\\))"
time="2025-04-07T10:12:07.947358658Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:c9b718423d7dfa8b4ac4f5d33e853621c044e9973942b98d799f70b4f418f71a err.message="blob unknown to registry" go.version=go1.20.8 http.request.host="sternum-registry-registry-server:5000" http.request.id=f767aed8-6222-4f7f-ae70-7277eb8a7fd1 http.request.method=HEAD http.request.remoteaddr="yy.yyy.yyy.yyy:49078" http.request.uri="/v2/timberio/vector/blobs/sha256:c9b718423d7dfa8b4ac4f5d33e853621c044e9973942b98d799f70b4f418f71a" http.request.useragent="docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.950767ms http.response.status=404 http.response.written=157 vars.digest="sha256:c9b718423d7dfa8b4ac4f5d33e853621c044e9973942b98d799f70b4f418f71a" vars.name="timberio/vector"
yy.yyy.yyy.yyy - - [07/Apr/2025:10:12:07 +0000] "HEAD /v2/timberio/vector/blobs/sha256:c9b718423d7dfa8b4ac4f5d33e853621c044e9973942b98d799f70b4f418f71a HTTP/1.0" 404 157 "" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \\(darwin\\))"
time="2025-04-07T10:12:07.948613214Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:dc8812030f8f6f0541690d058efbb028f4a0c67a1b2dc012e6745bd616bad930 err.message="blob unknown to registry" go.version=go1.20.8 http.request.host="sternum-registry-registry-server:5000" http.request.id=0d904391-9c4c-4617-b5ea-93a11a10f67e http.request.method=HEAD http.request.remoteaddr="yy.yyy.yyy.yyy:49092" http.request.uri="/v2/timberio/vector/blobs/sha256:dc8812030f8f6f0541690d058efbb028f4a0c67a1b2dc012e6745bd616bad930" http.request.useragent="docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.486712ms http.response.status=404 http.response.written=157 vars.digest="sha256:dc8812030f8f6f0541690d058efbb028f4a0c67a1b2dc012e6745bd616bad930" vars.name="timberio/vector"
yy.yyy.yyy.yyy - - [07/Apr/2025:10:12:07 +0000] "HEAD /v2/timberio/vector/blobs/sha256:dc8812030f8f6f0541690d058efbb028f4a0c67a1b2dc012e6745bd616bad930 HTTP/1.0" 404 157 "" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \\(darwin\\))"
time="2025-04-07T10:12:07.94961932Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 err.message="blob unknown to registry" go.version=go1.20.8 http.request.host="sternum-registry-registry-server:5000" http.request.id=5c9eb480-a34c-4695-8567-080e5ba10e83 http.request.method=HEAD http.request.remoteaddr="yy.yyy.yyy.yyy:49110" http.request.uri="/v2/timberio/vector/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" http.request.useragent="docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.37487ms http.response.status=404 http.response.written=157 vars.digest="sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" vars.name="timberio/vector"
yy.yyy.yyy.yyy - - [07/Apr/2025:10:12:07 +0000] "HEAD /v2/timberio/vector/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 HTTP/1.0" 404 157 "" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \\(darwin\\))"
time="2025-04-07T10:12:07.95060832Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:f771120c9c9899a957f14c0e7b131c26168deb7a9e5f7db2d365c27d42b7a8c3 err.message="blob unknown to registry" go.version=go1.20.8 http.request.host="sternum-registry-registry-server:5000" http.request.id=64858d9b-b189-47df-8e52-568ff847b169 http.request.method=HEAD http.request.remoteaddr="yy.yyy.yyy.yyy:49104" http.request.uri="/v2/timberio/vector/blobs/sha256:f771120c9c9899a957f14c0e7b131c26168deb7a9e5f7db2d365c27d42b7a8c3" http.request.useragent="docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.923978ms http.response.status=404 http.response.written=157 vars.digest="sha256:f771120c9c9899a957f14c0e7b131c26168deb7a9e5f7db2d365c27d42b7a8c3" vars.name="timberio/vector"
yy.yyy.yyy.yyy - - [07/Apr/2025:10:12:07 +0000] "HEAD /v2/timberio/vector/blobs/sha256:f771120c9c9899a957f14c0e7b131c26168deb7a9e5f7db2d365c27d42b7a8c3 HTTP/1.0" 404 157 "" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \\(darwin\\))"
time="2025-04-07T10:12:08.607512397Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="sternum-registry-registry-server:5000" http.request.id=af612105-3868-4eb0-82eb-96ac487c756a http.request.method=POST http.request.remoteaddr="yy.yyy.yyy.yyy:49122" http.request.uri="/v2/timberio/vector/blobs/uploads/" http.request.useragent="docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \(darwin\))" http.response.duration=8.435358ms http.response.status=202 http.response.written=0
yy.yyy.yyy.yyy - - [07/Apr/2025:10:12:08 +0000] "POST /v2/timberio/vector/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \\(darwin\\))"
yy.yyy.yyy.yyy - - [07/Apr/2025:10:12:08 +0000] "POST /v2/timberio/vector/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \\(darwin\\))"
time="2025-04-07T10:12:08.607707564Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="sternum-registry-registry-server:5000" http.request.id=f6955d0a-3c74-436d-98f3-761614c913d5 http.request.method=POST http.request.remoteaddr="yy.yyy.yyy.yyy:49114" http.request.uri="/v2/timberio/vector/blobs/uploads/" http.request.useragent="docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \(darwin\))" http.response.duration=8.766507ms http.response.status=202 http.response.written=0
time="2025-04-07T10:12:08.607623547Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="sternum-registry-registry-server:5000" http.request.id=055ea8c2-6a9d-43bc-a849-4d5e4e0923ce http.request.method=POST http.request.remoteaddr="yy.yyy.yyy.yyy:49134" http.request.uri="/v2/timberio/vector/blobs/uploads/" http.request.useragent="docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \(darwin\))" http.response.duration=7.4539ms http.response.status=202 http.response.written=0
yy.yyy.yyy.yyy - - [07/Apr/2025:10:12:08 +0000] "POST /v2/timberio/vector/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \\(darwin\\))"
time="2025-04-07T10:12:08.608641637Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="sternum-registry-registry-server:5000" http.request.id=7976b31b-d2e4-4616-8743-415edf1d5c85 http.request.method=POST http.request.remoteaddr="yy.yyy.yyy.yyy:49148" http.request.uri="/v2/timberio/vector/blobs/uploads/" http.request.useragent="docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \(darwin\))" http.response.duration=6.841978ms http.response.status=202 http.response.written=0
yy.yyy.yyy.yyy - - [07/Apr/2025:10:12:08 +0000] "POST /v2/timberio/vector/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \\(darwin\\))"
time="2025-04-07T10:12:08.625041248Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="sternum-registry-registry-server:5000" http.request.id=7b4dfc49-ec92-4213-a290-923234782893 http.request.method=POST http.request.remoteaddr="yy.yyy.yyy.yyy:49158" http.request.uri="/v2/timberio/vector/blobs/uploads/" http.request.useragent="docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \(darwin\))" http.response.duration=17.325549ms http.response.status=202 http.response.written=0
yy.yyy.yyy.yyy - - [07/Apr/2025:10:12:08 +0000] "POST /v2/timberio/vector/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \\(darwin\\))"
time="2025-04-07T10:12:14.479056962Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="sternum-registry-registry-server:5000" http.request.id=ae852185-56e3-47c7-b2e5-6a7bb7eeb456 http.request.method=POST http.request.remoteaddr="yy.yyy.yyy.yyy:57104" http.request.uri="/v2/timberio/vector/blobs/uploads/" http.request.useragent="docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \(darwin\))" http.response.duration=5.440862ms http.response.status=202 http.response.written=0
yy.yyy.yyy.yyy - - [07/Apr/2025:10:12:14 +0000] "POST /v2/timberio/vector/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \\(darwin\\))"
time="2025-04-07T10:12:14.488202508Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="sternum-registry-registry-server:5000" http.request.id=f5e8a5fe-78f8-43f1-96a5-23e3878999f7 http.request.method=POST http.request.remoteaddr="yy.yyy.yyy.yyy:57118" http.request.uri="/v2/timberio/vector/blobs/uploads/" http.request.useragent="docker/27.4.0 go/go1.22.10 git-commit/92a8393 kernel/6.10.14-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.4.0 \(darwin\))" http.response.duration=5.635118ms http.response.status=202 http.response.written=0

Any Idea what should I do to make it finally work? Would appreciate any help, thanks.

Apr 07 '25 10:04 itayvolo

Small progress… In one of the past issues, someone talked about validating that the certificate is set as CA (isCA: TRUE). Had to add it to the certificate's manifest and generate a new one (via cert-manager). Remounted it on the registry-ui pod and also copied the ca.crt to the docker client's certificates path.

now the error I get is:

tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "*.us-west-2.compute.amazonaws.com")

Apr 07 '25 16:04 itayvolo

Hi, thank you for using my project and submitting issues.

Sorry I will not be able to troubleshot your issue, configuration issues are taking to much time and I would like to be focus on the project itself.

I hope you will find the solution

Jun 29 '25 18:06 Joxit

Cannot push to HTTPS registry - k8s setup