k3s icon indicating copy to clipboard operation
k3s copied to clipboard

Published 20 hours ago • verified publisher icon k3s-io

Unable to make k3s's containerd use a private images registry

Open kCyborg opened this issue 3 years ago • 7 comments

Check at the end the edit, I have another problem now

Environmental Info: K3s Version:

k3s version v1.22.7+k3s1 (8432d7f2)
go version go1.16.10

Node(s) CPU architecture, OS, and Version:

amd64, Ubuntu, 20.04

Cluster Configuration:

Just one VM acting as ControlPlane+Worker.

Describe the bug: I'm working in a somehow isolated network, so I have configured my k3s's containerd registry to point a private docker proxy registry (which is a Harbor with a self signed certificate), following the official doc to allow my k3s node download all the necessary images to fulfill a completely successful installation, and also to let my future micro-services deployment download their images from my private docker registry previously mentioned. I can pull images from the harbor registry, both with docker pull and with /usr/local/bin/ctr -a /run/k3s/containerd/containerd.sock image pull , the command k3s crictl pull doesnt work at all tho. k3s doesn't notice "my order" to use my private registry and it goes to the wild internet to seek the images.

I guess I'm doing some kind of airgap installation if you guide by the Private Registry Method.

Steps To Reproduce:

  • Installed K3s following these steps: Just configuring the /etc/hosts file:
cat << EOF > /etc/hosts 
127.0.0.1 localhost
192.168.10.200 control01 control01.local 
10.10.0.19 harbor.virtalus.com
EOF
  • I pass the self signed certificates which I created in the Harbor VM while aprovisioning the service:
cat << EOF > /root/harbor.virtalus.com.crt
# [Here I pass the crt ]
EOF

cat << EOF > /root/harbor.virtalus.com.key
# [Here I pass the key ]
EOF

cat << EOF > /root/harbor.virtalus.com.cert
# [Here I pass the cert ]
EOF
  • I create the file from where containerd will get the mirror registry, where I pass the path to the certificates:
mkdir -p /etc/rancher/k3s/ && \
cat << EOF > /etc/rancher/k3s/registries.yaml
mirrors:
  docker.io:
    endpoint:
      - "https://harbor.virtalus.com/dockerhub/"
  "":
    endpoint:
      - "https://harbor.virtalus.com/dockerhub/"
  "*":
    endpoint:
      - "https://harbor.virtalus.com/dockerhub/"
configs:
  "harbor.virtalus.com":
    tls:
      cert_file: "/root/harbor.virtalus.com.cert"
      key_file: "/root/harbor.virtalus.com.key"
      ca_file: "/root/harbor.virtalus.com.crt"
      insecure_skip_verify: true
EOF
  • And then I install k3s, after the installation I can get without a problem kubectl get nodes and such commands:
curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -s - server

Expected behavior: containerd/k3s be able to pull the container images from my private self signed docker proxy registry (I use Harbor), ideally from the very begining of the installation.

Actual behavior: containerd/k3s is NOT able to pull the container images from my private self signed docker proxy registry, it keeps going to the internet to find those images.

Additional context / logs:

  • I'm perfectly able to, from the VM where I'm trying to install k3s, to reach my private registry both from IP and FQDN, and pull images from it manually from both proxies (in Harbor I created one pointing to dockerhub and another one pointing to quay.io):
/usr/local/bin/ctr -a /run/k3s/containerd/containerd.sock image pull harbor.virtalus.com/dockerhub/library/ubuntu:20.04 -k
harbor.virtalus.com/dockerhub/library/ubuntu:20.04:                               resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:115822d64890aae5cde3c1e85ace4cc97308bb1fd884dac62f4db0a16dbddb36:    exists         |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:7b3e30a1f373b0621681f13b92feb928129c1c38977481ee788a793fcae64fb9: exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8e5c1b329fe39c318c0d49821b339fb94a215c5dc0a2898c8030b5a4d091bcba:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:1a437e363abfa47bfe4b3f5906b7444d12346102d944ebddd537e47a62fc6f52:   exists         |++++++++++++++++++++++++++++++++++++++|
elapsed: 71.8s                                                                    total:  27.2 M (388.5 KiB/s)
unpacking linux/amd64 sha256:115822d64890aae5cde3c1e85ace4cc97308bb1fd884dac62f4db0a16dbddb36...
done: 1.772760742s

/usr/local/bin/ctr -a /run/k3s/containerd/containerd.sock image pull harbor.virtalus.com/q/metallb/speaker:v0.12.1 -k
harbor.virtalus.com/q/metallb/speaker:v0.12.1:                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:40e0373788f3ea5e21a42b302a0ae4fd537646eb4b485e02b7ae52dc71da230c:    exists         |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:f4504b24983c61ea2fd6fa185f8b5ac870a303e04e0ccdcdb099e3fca90b1447: exists         |++++++++++++++++++++++++++++++++++++++|
config-sha256:579ce8a43ea820ef260ede80522cf92c59fd0a59f97666699aa5b878e75a4ccf:   exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:b3c9c7ec0554016a59eff220003aa5b3e2f6660251cc4f9fbb12e7b0ee876273:    exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8f5232735740bcca8ce70fd5c429801612f51d566e2abf44726248e9586037d5:    downloading    |--------------------------------------|    0.0 B/18.3 MiB
layer-sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3:    downloading    |--------------------------------------|    0.0 B/2.7 MiB
layer-sha256:c914caf2a709e2095fc85de82d195c297f65f39cecaac6f59f5047243ceb5b33:    downloading    |--------------------------------------|    0.0 B/8.2 MiB
elapsed: 7.1 s                                                                    total:   0.0 B (0.0 B/s)
  • If I check the /var/lib/rancher/k3s/agent/etc/containerd/config.toml then I get the following, which makes me think it's correctly configured:
cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml

[plugins.opt]
  path = "/var/lib/rancher/k3s/agent/containerd"

[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins.cri.containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins.cri.cni]
  bin_dir = "/var/lib/rancher/k3s/data/31ff0fd447a47323a7c863dbb0a3cd452e12b45f1ec67dc55efa575503c2c3ac/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"

[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins.cri.registry.mirrors]

[plugins.cri.registry.mirrors.""]
  endpoint = ["https://harbor.virtalus.com/dockerhub/"]

[plugins.cri.registry.mirrors."*"]
  endpoint = ["https://harbor.virtalus.com/dockerhub/"]

[plugins.cri.registry.mirrors."docker.io"]
  endpoint = ["https://harbor.virtalus.com/dockerhub/"]

[plugins.cri.registry.configs."harbor.virtalus.com".tls]
  ca_file = "/root/harbor.virtalus.com.crt"
  cert_file = "/root/harbor.virtalus.com.cert"
  key_file = "/root/harbor.virtalus.com.key"
  insecure_skip_verify = true
  • But when I try with k3s crictl pull (which I think is actually the command that runs under the hood while k3s tries to pull an image) I get the following error:
 k3s crictl pull harbor.virtalus.com/q/metallb/speaker:v0.12.1
FATA[0002] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "harbor.virtalus.com/q/metallb/speaker:v0.12.1": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found
  • And when I check the containerd's logs it keeps telling me it wanna go for docker.io while I'm explicitly telling it to use harbor.virtalus.com:
systemctl restart k3s
tail -f /var/lib/rancher/k3s/agent/containerd/containerd.log
time="2022-04-23T07:20:10.079082231-04:00" level=info msg="Start recovering state"
time="2022-04-23T07:20:10.079254378-04:00" level=warning msg="The image content readiness for \"docker.io/rancher/mirrored-pause:3.6\" is not ok"
time="2022-04-23T07:20:10.079314031-04:00" level=info msg=serving... address=/run/k3s/containerd/containerd.sock.ttrpc
time="2022-04-23T07:20:10.079383141-04:00" level=info msg=serving... address=/run/k3s/containerd/containerd.sock
time="2022-04-23T07:20:10.079409022-04:00" level=info msg="containerd successfully booted in 0.036387s"
time="2022-04-23T07:20:10.084748475-04:00" level=info msg="Start event monitor"
time="2022-04-23T07:20:10.084790778-04:00" level=info msg="Start snapshots syncer"
time="2022-04-23T07:20:10.084803180-04:00" level=info msg="Start cni network conf syncer"
time="2022-04-23T07:20:10.084815390-04:00" level=info msg="Start streaming server"
time="2022-04-23T07:20:15.912042279-04:00" level=info msg="No cni config template is specified, wait for other system components to drop the config."
time="2022-04-23T07:20:17.695301845-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:local-path-provisioner-84bb864455-xk4qn,Uid:d33c0d0c-23ba-4cf0-af9a-e21dd7450a99,Namespace:kube-system,Attempt:0,}"
time="2022-04-23T07:20:17.994951495-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:metrics-server-ff9dbcb6c-ppqqn,Uid:61b8f09f-9eda-4138-b207-cc7f9b4a43b4,Namespace:kube-system,Attempt:0,}"
time="2022-04-23T07:20:17.994988204-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:helm-install-traefik-crd--1-zlsck,Uid:bb0e923b-9360-4980-9b4a-6eec19b758d3,Namespace:kube-system,Attempt:0,}"
time="2022-04-23T07:20:18.295300672-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:coredns-96cc4f57d-szcx5,Uid:02f2fd91-3ff3-49d0-8d73-9656730b5f0b,Namespace:kube-system,Attempt:0,}"
time="2022-04-23T07:20:18.594658520-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:helm-install-traefik--1-9g4n9,Uid:4882bb74-f80a-4ece-a4df-b7fad63ab11e,Namespace:kube-system,Attempt:0,}"
time="2022-04-23T07:20:19.207755114-04:00" level=warning msg="reference for unknown type: text/html" digest="sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21" mediatype=text/html size=906
time="2022-04-23T07:20:19.287099519-04:00" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/rancher/mirrored-pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2022-04-23T07:20:19.287364892-04:00" level=error msg="Failed to handle event &ImageUpdate{Name:docker.io/rancher/mirrored-pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],} for docker.io/rancher/mirrored-pause:3.6" error="update image store for \"docker.io/rancher/mirrored-pause:3.6\": get image info from containerd: get image diffIDs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found"
time="2022-04-23T07:20:19.335051527-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:local-path-provisioner-84bb864455-xk4qn,Uid:d33c0d0c-23ba-4cf0-af9a-e21dd7450a99,Namespace:kube-system,Attempt:0,} failed, error" error="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found"
time="2022-04-23T07:20:19.549246084-04:00" level=warning msg="reference for unknown type: text/html" digest="sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21" mediatype=text/html size=906
time="2022-04-23T07:20:19.658608686-04:00" level=info msg="Events for \"docker.io/rancher/mirrored-pause:3.6\" is in backoff, enqueue event &ImageUpdate{Name:docker.io/rancher/mirrored-pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2022-04-23T07:20:19.681038462-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:helm-install-traefik-crd--1-zlsck,Uid:bb0e923b-9360-4980-9b4a-6eec19b758d3,Namespace:kube-system,Attempt:0,} failed, error" error="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found"
time="2022-04-23T07:20:19.790246872-04:00" level=warning msg="reference for unknown type: text/html" digest="sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21" mediatype=text/html size=906
time="2022-04-23T07:20:19.794056869-04:00" level=warning msg="reference for unknown type: text/html" digest="sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21" mediatype=text/html size=906
time="2022-04-23T07:20:19.936894023-04:00" level=info msg="Events for \"docker.io/rancher/mirrored-pause:3.6\" is in backoff, enqueue event &ImageUpdate{Name:docker.io/rancher/mirrored-pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2022-04-23T07:20:19.962206072-04:00" level=info msg="Events for \"docker.io/rancher/mirrored-pause:3.6\" is in backoff, enqueue event &ImageUpdate{Name:docker.io/rancher/mirrored-pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2022-04-23T07:20:19.990579053-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-96cc4f57d-szcx5,Uid:02f2fd91-3ff3-49d0-8d73-9656730b5f0b,Namespace:kube-system,Attempt:0,} failed, error" error="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found"
time="2022-04-23T07:20:20.008587594-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:metrics-server-ff9dbcb6c-ppqqn,Uid:61b8f09f-9eda-4138-b207-cc7f9b4a43b4,Namespace:kube-system,Attempt:0,} failed, error" error="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found"
time="2022-04-23T07:20:20.083796940-04:00" level=warning msg="reference for unknown type: text/html" digest="sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21" mediatype=text/html size=906
time="2022-04-23T07:20:20.156032946-04:00" level=info msg="Events for \"docker.io/rancher/mirrored-pause:3.6\" is in backoff, enqueue event &ImageUpdate{Name:docker.io/rancher/mirrored-pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2022-04-23T07:20:20.173171864-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:helm-install-traefik--1-9g4n9,Uid:4882bb74-f80a-4ece-a4df-b7fad63ab11e,Namespace:kube-system,Attempt:0,} failed, error" error="rpc error: code = NotFound desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found"
time="2022-04-23T07:20:21.085809971-04:00" level=info msg="ImageUpdate event &ImageUpdate{Name:docker.io/rancher/mirrored-pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
time="2022-04-23T07:20:21.085950986-04:00" level=error msg="Failed to handle backOff event &ImageUpdate{Name:docker.io/rancher/mirrored-pause:3.6,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],} for docker.io/rancher/mirrored-pause:3.6" error="update image store for \"docker.io/rancher/mirrored-pause:3.6\": get image info from containerd: get image diffIDs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found"

Backporting I don't really know if this issue needs backporting.

Another problem I was able to "fix" the previous problem by doing a fully airgap installation, but when I try to:

 k3s crictl pull harbor.virtalus.com/q/metallb/speaker:v0.12.1

I get the following error:

FATA[0002] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "harbor.virtalus.com/q/metallb/speaker:v0.12.1": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found

Although I'm still able to:

ctr i pull -k harbor.virtalus.com/q/metallb/speaker:v0.12.1
harbor.virtalus.com/q/metallb/speaker:v0.12.1:                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:40e0373788f3ea5e21a42b302a0ae4fd537646eb4b485e02b7ae52dc71da230c:    exists         |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:f4504b24983c61ea2fd6fa185f8b5ac870a303e04e0ccdcdb099e3fca90b1447: exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3:    exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c914caf2a709e2095fc85de82d195c297f65f39cecaac6f59f5047243ceb5b33:    downloading    |++++++++++++++++++--------------------|  4.0 MiB/8.2 MiB
layer-sha256:b3c9c7ec0554016a59eff220003aa5b3e2f6660251cc4f9fbb12e7b0ee876273:    exists         |++++++++++++++++++++++++++++++++++++++|
config-sha256:579ce8a43ea820ef260ede80522cf92c59fd0a59f97666699aa5b878e75a4ccf:   exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8f5232735740bcca8ce70fd5c429801612f51d566e2abf44726248e9586037d5:    downloading    |++++++++++++++------------------------|  7.0 MiB/18.3 MiB
elapsed: 4.8 s                                                                    total:  11.0 M (2.3 MiB/s)



/usr/local/bin/ctr -a /run/k3s/containerd/containerd.sock image pull harbor.virtalus.com/dockerhub/library/ubuntu:20.04 -k
harbor.virtalus.com/dockerhub/library/ubuntu:20.04:                               resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:115822d64890aae5cde3c1e85ace4cc97308bb1fd884dac62f4db0a16dbddb36:    exists         |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:7b3e30a1f373b0621681f13b92feb928129c1c38977481ee788a793fcae64fb9: exists         |++++++++++++++++++++++++++++++++++++++|
config-sha256:1a437e363abfa47bfe4b3f5906b7444d12346102d944ebddd537e47a62fc6f52:   exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8e5c1b329fe39c318c0d49821b339fb94a215c5dc0a2898c8030b5a4d091bcba:    downloading    |--------------------------------------|    0.0 B/27.2 MiB
elapsed: 3.3 s                                                                    total:   0.0 B (0.0 B/s)

kCyborg avatar Apr 23 '22 11:04 kCyborg

There are a couple problems with what you're doing.

  1. You've set the endpoint for your registry mirror tohttps://harbor.virtalus.com/dockerhub/. This does not mean that all image docker.io/x will be pulled as harbor.virtalus.com/dockerhub/x, which is what I think you want. What you're actually doing is setting the root of the Docker V2 registry api to /dockerhub, which explains why you're getting text/html responses when pulling images. You're probably getting an error page from your Harbor registry. What you probably want is this:
mirrors:
  docker.io:
    endpoint:
      - "https://harbor.virtalus.com/v2"
    rewrite:
      "(.*)": "dockerhub/$1"
configs:
  "harbor.virtalus.com":
    tls:
      insecure_skip_verify: true
  1. When you configure a registry endpoint, this is only used internally as the location to pull the image from. It DOES NOT alter the name of the image that is pulled. If you pull docker.io/x, it will always only show up as docker.io/x when pulled, even if it came from your mirror.
  2. Containerd will ALWAYS fall back to the default endpoint (docker.io in this case) if it can't get the image from your mirror. The only way to disable this is to block access to the upstream registry at the network level.

brandond avatar Apr 25 '22 22:04 brandond

what if I have an authorization for my registry? I wrote things into my /etc/rancher/k3s/registries.yaml like:

mirrors:
  myregistry:
    endpoint:
      - "http://myregistry"
configs:
  "myregistry":
    auth:
      username: myusr
      password: mypasswd

and I restarted k3s as well as containerd here I tried k3s crictl pull --creds myusr:mypasswd myregistry/myimage:v1, it worked, but k3s crictl pull myregistry/myimage:v1 failed, unsuperisingly, when k3s kubectl creates something, it cannot pull it either. Seems that the /etc/rancher/k3s/registries.yaml didn't work, what should I check?

lidh15 avatar Jul 28 '22 17:07 lidh15

what if I have an authorization for my registry? I wrote things into my /etc/rancher/k3s/registries.yaml like:

mirrors:
  myregistry:
    endpoint:
      - "http://myregistry"
configs:
  "myregistry":
    auth:
      username: myusr
      password: mypasswd

and I restarted k3s as well as containerd here I tried k3s crictl pull --creds myusr:mypasswd myregistry/myimage:v1, it worked, but k3s crictl pull myregistry/myimage:v1 failed, unsuperisingly, when k3s kubectl creates something, it cannot pull it either. Seems that the /etc/rancher/k3s/registries.yaml didn't work, what should I check?

I'm using cri-docker and docker rather than containerd sothere is no config.toml. I wonder if there is a counterpart?

lidh15 avatar Jul 28 '22 19:07 lidh15

If you're using your own container runtime, you need to configure registry mirrors manually. k3s's registries.yaml is only used to configure the embedded containerd.

brandond avatar Jul 28 '22 21:07 brandond

If you're using your own container runtime, you need to configure registry mirrors manually. k3s's registries.yaml is only used to configure the embedded containerd.

would you please offer more detailslike which file to edit? to the best of my knowledge 'configure registry mirrors' means 'docker login' and it has worked, docker do logged things in ~/.docker/, but crictl pull won't use that information.

lidh15 avatar Jul 29 '22 00:07 lidh15

we don't have crictl login, right? does that mean it is impossible to do kubectl create without locally present images and I have to crictl pull --creds first?

lidh15 avatar Jul 29 '22 00:07 lidh15

@lidh15 if you're managing your own containerd, I would defer to the containerd docs: https://github.com/containerd/containerd/blob/main/docs/cri/registry.md

brandond avatar Aug 01 '22 21:08 brandond

just wanted to say thanks --- i had exact same issue -- harbor needs that /v2 at the end.

hlacikd avatar Nov 03 '22 16:11 hlacikd

For 99.9% of repos, you either need to include the /v2 at the end, or to omit the path entirely and just stop at the hostname:port (in which case containerd adds the correct path for you).

brandond avatar Nov 03 '22 17:11 brandond