nerdctl icon indicating copy to clipboard operation
nerdctl copied to clipboard

Published 20 hours ago • verified publisher icon containerd

nerdctl pull from private registry mirror is missing Authorization header

Open networkhermit opened this issue 11 months ago • 0 comments

Description

I tried to use nerdctl with registry mirrors served by private harbor, but I got 401 Unauthorized error.

After some investigation, I found that when using registry mirrors, nerdctl doesn't pass the Authorization: Basic xxxx header to /service/token endpoints, causing the final 401 Unauthorized error.

Steps to reproduce the issue

  1. configure containerd to use registry mirrors:

/etc/containerd/config.toml

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"

/etc/containerd/certs.d/gcr.io/hosts.toml

server = "https://gcr.io"

[host."https://harbor.example.com:8443/v2/gcr.io"]
  capabilities = ["pull", "resolve"]
  override_path = true

  1. use nerdctl login to harbor.example.com:8443, optionally setup mitmproxy to view the api traffic

  2. directly pull the image from harbor is successfully

❯ sudo HTTPS_PROXY=http://127.0.0.1:8080 nerdctl --debug-full pull harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest
DEBU[0000] verifying process skipped
DEBU[0000] Found hosts dir "/etc/containerd/certs.d"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs".
DEBU[0000] fetching                                      image="harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest"
DEBU[0000] resolving                                     host="harbor.example.com:8443"
DEBU[0000] do request                                    host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest"
harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest: resolving      |--------------------------------------|
elapsed: 0.3 s                                             total:   0.0 B (0.0 B/s)
DEBU[0000] fetch response received                       host="harbor.example.com:8443" response.header.content-length=152 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Feb 2024 14:05:20 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=istio-envoy response.header.set-cookie="sid=7bc1d436bafba2a71e19ec32cfcc06b8; Path=/; HttpOnly" response.header.www-authenticate="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" response.header.x-envoy-upstream-service-time=7 response.header.x-request-id=cb37ff75-4c43-4d62-a565-2d767adbd4d8 response.status="401 Unauthorized" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" host="harbor.example.com:8443"
DEBU[0000] do request                                    host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.inde
harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest: resolving      |--------------------------------------|
elapsed: 2.0 s                                             total:   0.0 B (0.0 B/s)
DEBU[0002] fetch response received                       host="harbor.example.com:8443" response.header.content-length=1163 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Tue, 27 Feb 2024 14:05:22 GMT" response.header.docker-content-digest="sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8\"" response.header.server=istio-envoy response.header.set-cookie="sid=1fbb1be72a6e2fd04a4962cfd6997843; Path=/; HttpOnly" response.header.x-envoy-upstream-service-time=1660 response.header.x-request-id=6dfd0fa9-f711-4512-a26f-ed627a26ab73 response.status="200 OK" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest"
DEBU[0002] resolved                                      desc.digest="sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8" host="harbor.example.com:8443"
DEBU[0002] fetch                                         digest="sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1163
DEBU[0002] fetch                                         digest="sha256:68c29634fe49724f94ed34f18224316f776392f7a5a4014969ac5798a2ec96dc" mediatype=application/vnd.docker.container.image.v1+json size=4843
DEBU[0002] layer unpacked                                duration="221.474µs" layer="sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964"
DEBU[0002] layer unpacked                                duration="87.424µs" layer="sha256:2d4828968d6ca90e549ac054b8330c28081e02b31574f663fe8dae9c03275103"
DEBU[0002] layer unpacked                                duration="62.677µs" layer="sha256:b28a9b13dc6dccb175978a3e34f0ee5ec435e1328b9ded306724e490388585e4"
DEBU[0002] layer unpacked                                duration="92.493µs" layer="sha256:031224e6222c780a9783d9d287f6c25e37f16c800ba4bbb13e7f9c77036fb811"
harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest:                        resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:68c29634fe49724f94ed34f18224316f776392f7a5a4014969ac5798a2ec96dc:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.2 s                                                                    total:   0.0 B (0.0 B/s)
  1. pulling the same image from registry mirrors fails as 401
❯ sudo HTTPS_PROXY=127.0.0.1:8080 nerdctl --debug-full pull gcr.io/cadvisor/cadvisor:latest
DEBU[0000] verifying process skipped
DEBU[0000] Found hosts dir "/etc/containerd/certs.d"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs".
DEBU[0000] fetching                                      image="gcr.io/cadvisor/cadvisor:latest"
DEBU[0000] loading host directory                        dir=/etc/containerd/certs.d/gcr.io
DEBU[0000] resolving                                     host="harbor.example.com:8443"
DEBU[0000] do request                                    host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] fetch response received                       host="harbor.example.com:8443" response.header.content-length=152 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Feb 2024 03:27:43 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=istio-envoy response.header.set-cookie="sid=63cc905e0a57374c5292b8273645d06c; Path=/; HttpOnly" response.header.www-authenticate="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" response.header.x-envoy-upstream-service-time=3 response.header.x-request-id=d0e29292-1e9e-4f15-ab65-51d37f1dd851 response.status="401 Unauthorized" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" host="harbor.example.com:8443"
DEBU[0000] do request                                    host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] fetch response received                       host="harbor.example.com:8443" response.header.content-length=198 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Feb 2024 03:27:43 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=istio-envoy response.header.set-cookie="sid=5ce41f88f2ba4e9246430cfa8e46312f; Path=/; HttpOnly" response.header.www-authenticate="Basic realm=\"harbor\"" response.header.x-envoy-upstream-service-time=1 response.header.x-request-id=0c1dc612-bbaa-4638-ba67-18f0a69006cf response.status="401 Unauthorized" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] Unauthorized                                  header="Basic realm=\"harbor\"" host="harbor.example.com:8443"
DEBU[0000] resolving                                     host=gcr.io
DEBU[0000] do request                                    host=gcr.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://gcr.io/v2/cadvisor/cadvisor/manifests/latest"
gcr.io/cadvisor/cadvisor:latest: resolving      |--------------------------------------|
elapsed: 133.1s                  total:   0.0 B (0.0 B/s)
INFO[0133] trying next host                              error="failed to do request: Head \"https://gcr.io/v2/cadvisor/cadvisor/manifests/latest\": Bad Gateway" host=gcr.io
FATA[0133] failed to resolve reference "gcr.io/cadvisor/cadvisor:latest": unexpected status from HEAD request to https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io: 401 Unauthorized
  1. Browse through the mitmproxy api requests, I can see the request tohttps://harbor.example.com:8443/service/token?scope=repository%3Acadvisor%2Fcadvisor%3Apull&scope=repository%3Agcr.io%2Fcadvisor%2Fcadvisor%3Apull&service=harbor-registry lacks the Authorization: Basic xxxx header, hence results the final 401 error.

I also checked the mitmproxy api traffic using ctr and crictl, on successfully image pulling from registry mirrors, the above Authorization: Basic xxxx header is both present.

Describe the results you received and expected

nerdctl pull from private registry mirror works, the credential of the mirror registry can be passed and reused from nerdctl login credential and hence adding the missing Authorization: Basic xxxx header.

What version of nerdctl are you using?

Client: Version: 1.7.2 OS/Arch: linux/amd64 Git commit: e32c4b023bf41e5c8325cfb893a53cefb5fc68ed buildctl: Version:

Server: containerd: Version: v1.7.13 GitCommit: 7c3aca7a610df76212171d200ca3811ff6096eb8.m runc: Version: 1.1.12

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

Client: Namespace: default Debug Mode: false

Server: Server Version: v1.7.13 Storage Driver: overlayfs Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Log: fluentd journald json-file syslog Storage: native overlayfs Security Options: seccomp Profile: builtin cgroupns Kernel Version: 6.7.6-arch1-1 Operating System: Arch Linux OSType: linux Architecture: x86_64 CPUs: 32 Total Memory: 125GiB Name: arch ID: 36b0778a-4ce8-4dca-af98-5afa21440138

networkhermit avatar Feb 27 '24 15:02 networkhermit