[BUG] k3d uses https to ask images from an http private registry

[fragolinux]

I use k3d 5.4.3 to create a cluster with: k3d cluster create --config cluster.yaml

my cluster.yaml is this (all values are correctly substituted from env vars by k3d itself):

# info: https://k3d.io/v5.4.3/usage/configfile/
apiVersion: k3d.io/v1alpha4 # this will change in the future as we make everything more stable
kind: Simple # internally, we also have a Cluster config, which is not yet available externally
metadata:
  name: $CLUSTER_NAME # name that you want to give to your cluster (will still be prefixed with `k3d-`)
kubeAPI: # same as `--api-port myhost.my.domain:6445` (where the name would resolve to 127.0.0.1)
  host: $LOCALDOMAIN # important for the `server` setting in the kubeconfig
  hostIP: "127.0.0.1" # where the Kubernetes API will be listening on
  hostPort: "6445" # where the Kubernetes API listening port will be mapped to on your host system
image: rancher/$IMAGE # same as `--image rancher/k3s:v1.20.4-k3s1`
volumes: # repeatable flags are represented as YAML lists
  - volume: $DOMAIN_STORAGE:/var/lib/rancher/k3s/storage # same as `--volume '/my/host/path:/path/in/node@server:0;agent:*'`
    nodeFilters:
      - all
ports:
  - port: 80:80 # same as `--port '80:80@loadbalancer'`
    nodeFilters:
      - loadbalancer
  - port: 443:443 # same as `--port '443:443@loadbalancer'`
    nodeFilters:
      - loadbalancer
registries: # define how registries should be created or used
  create: # creates a default registry to be used with the cluster; same as `--registry-create registry.localhost`
    name: local-registry
    host: "0.0.0.0"
    hostPort: "5555"
    volumes:
      - $regDir:/var/lib/registry # persist registry data locally
  config: | # define contents of the `registries.yaml` file (or reference a file); same as `--registry-config /path/to/config.yaml`
    mirrors:
      $LOCALDOMAIN:
        endpoint:
          - "http://$LOCALDOMAIN:5555"
hostAliases: # /etc/hosts style entries to be injected into /etc/hosts in the node containers and in the NodeHosts section in CoreDNS
  - ip: $PUBLICIP
    hostnames:
      - $LOCALDOMAIN
options:
  k3d: # k3d runtime settings
    wait: true # wait for cluster to be usable before returining; same as `--wait` (default: true)
    timeout: "60s" # wait timeout before aborting; same as `--timeout 60s`
  k3s: # options passed on to K3s itself
    extraArgs: # additional arguments passed to the `k3s server|agent` command; same as `--k3s-arg`
      - arg: --tls-san=$LOCALDOMAIN
        nodeFilters:
          - server:0
      - arg: --disable=traefik
        nodeFilters:
          - server:0
  kubeconfig:
    updateDefaultKubeconfig: true # add new cluster to your default Kubeconfig; same as `--kubeconfig-update-default` (default: true)
    switchCurrentContext: true # also set current-context to the new cluster's context; same as `--kubeconfig-switch-context` (default: true)

I can pull and push images to the local registry, no problems, I can even run local docker containers from that image

docker pull nginx:latest
docker tag nginx:latest e4t.local:5555/nginx:latest
docker push e4t.local:5555/nginx:latest
docker run -d --name=mynginx e4t.local:5555/nginx:latest # works
kubectl run mynginx --image e4t.local:5555/nginx:latest # NOT working

last command reports this in logs:

E0628 20:32:07.266829 6 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"mynginx\" with ErrImagePull: \"rpc error: code = Unknown desc = failed to pull and unpack image \\\"e4t.local:5555/nginx:latest\\\": failed to resolve reference \\\"e4t.local:5555/nginx:latest\\\": failed to do request: Head \\\"https://e4t.local:5555/v2/nginx/manifests/latest\\\": http: server gave HTTP response to https client\"" pod="default/mynginx" podUID=cd188aa2-aa51-4e40-9e48-bfd709225fd6

so, something wrong, k3d is searching for a registry in https while mine is in http (my docker contains the line "insecure-registries": [ "e4t.local:5555"] )

I tried to switch to https then:

# info: https://k3d.io/v5.4.3/usage/configfile/
apiVersion: k3d.io/v1alpha4 # this will change in the future as we make everything more stable
kind: Simple # internally, we also have a Cluster config, which is not yet available externally
metadata:
  name: $CLUSTER_NAME # name that you want to give to your cluster (will still be prefixed with `k3d-`)
kubeAPI: # same as `--api-port myhost.my.domain:6445` (where the name would resolve to 127.0.0.1)
  host: $LOCALDOMAIN # important for the `server` setting in the kubeconfig
  hostIP: "127.0.0.1" # where the Kubernetes API will be listening on
  hostPort: "6445" # where the Kubernetes API listening port will be mapped to on your host system
image: rancher/$IMAGE # same as `--image rancher/k3s:v1.20.4-k3s1`
volumes: # repeatable flags are represented as YAML lists
  - volume: $DOMAIN_STORAGE:/var/lib/rancher/k3s/storage # same as `--volume '/my/host/path:/path/in/node@server:0;agent:*'`
    nodeFilters:
      - all
ports:
  - port: 80:80 # same as `--port '80:80@loadbalancer'`
    nodeFilters:
      - loadbalancer
  - port: 443:443 # same as `--port '443:443@loadbalancer'`
    nodeFilters:
      - loadbalancer
registries: # define how registries should be created or used
  create: # creates a default registry to be used with the cluster; same as `--registry-create registry.localhost`
    name: local-registry
    host: "0.0.0.0"
    hostPort: "5555"
    volumes:
      - $regDir:/var/lib/registry # persist registry data locally
      - $HOME/.local/share/mkcert/rootCA.pem:/etc/ssl/certs/rootCA.pem
      - $caDir/localCert.pem:/etc/ssl/certs/localCert.pem
      - $caDir/localKey.pem:/etc/ssl/certs/localKey.pem
  config: | # define contents of the `registries.yaml` file (or reference a file); same as `--registry-config /path/to/config.yaml`
    mirrors:
      $LOCALDOMAIN:
        endpoint:
          - "https://$LOCALDOMAIN:5555"
    configs:
      $LOCALDOMAIN:
        tls:
          ca_file: "/etc/ssl/certs/rootCA.pem"
          cert_file: "/etc/ssl/certs/localCert.pem"
          key_file: "/etc/ssl/certs/localKey.pem"
hostAliases: # /etc/hosts style entries to be injected into /etc/hosts in the node containers and in the NodeHosts section in CoreDNS
  - ip: $PUBLICIP
    hostnames:
      - $LOCALDOMAIN
options:
  k3d: # k3d runtime settings
    wait: true # wait for cluster to be usable before returining; same as `--wait` (default: true)
    timeout: "60s" # wait timeout before aborting; same as `--timeout 60s`
  k3s: # options passed on to K3s itself
    extraArgs: # additional arguments passed to the `k3s server|agent` command; same as `--k3s-arg`
      - arg: --tls-san=$LOCALDOMAIN
        nodeFilters:
          - server:0
      - arg: --disable=traefik
        nodeFilters:
          - server:0
  kubeconfig:
    updateDefaultKubeconfig: true # add new cluster to your default Kubeconfig; same as `--kubeconfig-update-default` (default: true)
    switchCurrentContext: true # also set current-context to the new cluster's context; same as `--kubeconfig-switch-context` (default: true)

but with such setup, I get http answers from the https url!

curl https://e4t.local:5555/v2/_catalog
curl: (35) error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version

what's wrong in this config? All variables and cert files are in the right position (I went into the pod to check under /etc and they're there)

[fragolinux]

I can get a working registry with this setup, using an external docker container:

docker run -d \
  --restart=always \
  --name registry \
  -v $caDir/localCert.pem:/etc/ssl/certs/localCert.pem \
  -v $caDir/localKey.pem:/etc/ssl/certs/localKey.pem \
  -v $(pwd)/registry:/var/lib/registry \
  -e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/ssl/certs/localCert.pem \
  -e REGISTRY_HTTP_TLS_KEY=/etc/ssl/certs/localKey.pem \
  -p 5555:443 \
  registry:2