[BUG] Issue pulling image from k3d managed local registry

[markoelez]

Issue pulling image from k3d managed local registry

• How was the cluster created?

created registry:

k3d registry create registry.localhost --port 8083

created registry.yaml:

mirrors:
  'localhost:8083':
    endpoint:
      - http://k3d-registry.localhost:8083

created cluster:

k3d cluster create dev -p "8000:80@loadbalancer" --registry-use k3d-registry.localhost:8083 --registry-config registry.yaml

run test:

docker pull nginx
docker tag nginx:latest localhost:8083/nginx:latest
docker push localhost:8083/nginx:latest

kubectl run nginx --image localhost:8083/nginx:latest

output:

Using default tag: latest
latest: Pulling from library/nginx
Digest: sha256:943c25b4b66b332184d5ba6bb18234273551593016c0e0ae906bab111548239f
Status: Image is up to date for nginx:latest
docker.io/library/nginx:latest
The push refers to repository [localhost:8087/nginx]
a2e59a79fae0: Pushed 
4091cd312f19: Pushed 
9e7119c28877: Pushed 
2280b348f4d6: Pushed 
e74d0d8d2def: Pushed 
a12586ed027f: Pushed 
latest: digest: sha256:06aa2038b42f1502b59b3a862b1f5980d3478063028d8e968f0810b9b0502380 size: 1570
pod/nginx created

What did you expect to happen

I expected a running nginx pod in my default name space to spin up.

Screenshots or terminal output

However, I got the following:

...
Volumes:
  kube-api-access-r7g65:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  99s                default-scheduler  Successfully assigned default/nginx to k3d-dev-server-0
  Normal   Pulling    16s (x4 over 99s)  kubelet            Pulling image "localhost:8087/nginx:latest"
  Warning  Failed     16s (x4 over 99s)  kubelet            Failed to pull image "localhost:8087/nginx:latest": rpc error: code = Unknown desc = failed to pull and unpack image "localhost:8087/nginx:latest": failed to resolve reference "localhost:8087/nginx:latest": failed to do request: Head "http://k3d-registry.localhost:8087/v2/nginx/manifests/latest?ns=localhost%3A8087": dial tcp 172.22.0.2:8087: connect: connection refused
  Warning  Failed     16s (x4 over 99s)  kubelet            Error: ErrImagePull
  Normal   BackOff    5s (x5 over 99s)   kubelet            Back-off pulling image "localhost:8087/nginx:latest"
  Warning  Failed     5s (x5 over 99s)   kubelet            Error: ImagePullBackOff

Which OS & Architecture

arch: x86_64
cgroupdriver: cgroupfs
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
name: docker
os: Docker Desktop
ostype: linux
version: 20.10.17

Which version of k3d

k3d version v5.4.6
k3s version v1.24.4-k3s1 (default)

Which version of docker

version:

Client:
 Cloud integration: v1.0.28
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:04:45 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.11.1 (84025)
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:23 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2)
  compose: Docker Compose (Docker Inc., v2.7.0)
  extension: Manages Docker extensions (Docker Inc., v0.2.8)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 17
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc version: v1.1.2-0-ga916309
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.675GiB
 Name: docker-desktop
 ID: QJTY:3BRB:6RZ5:GCOG:3TLQ:XHXA:CM5T:XXX7:VGMS:XLNL:CIG5:EALJ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Username: marko7
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

[john-funk]

I was also struggling to get this to work.

In the end I got a working setup using the following config in the k3d.yaml:

...
registries:
  create:
    name: registry
    hostPort: "5000"
  config: |
    mirrors:
      "registry:5000":
        endpoint:
          - http://registry:5000

Then create the cluster with k3d cluster create mycluster --config k3d.yaml

Pushing to the registry only works using localhost:5000/<IMAGE_NAME>:<IMAGE_TAG> For Deployments in the cluster I have to specify registry:5000/<IMAGE_NAME>:<IMAGE_TAG>