k3s with private registry

[Drtuna]

Version: k3s version v0.8.1 (d116e74a) and k3s version v1.18.2+k3s1 (698e444a) K3s arguments: I ran k3s with docker compose as described in the manual with a minor change concerning TLS

version: '3.2'
services:
  server:
    image: rancher/k3s:latest
    command: server  --disable-agent --tls-san 192.168.2.110 
    environment:
    - K3S_CLUSTER_SECRET=somethingtotallyrandom
    - K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
    - K3S_KUBECONFIG_MODE=666
    volumes:
    - k3s-server:/var/lib/rancher/k3s
    # get the kubeconfig file
    - .:/output
    - ./registries.yaml:/etc/rancher/k3s/registries.yaml
    ports:
     - 192.168.2.110:6443:6443

  node:
    image: rancher/k3s:latest
    volumes:
    - ./registries.yaml:/etc/rancher/k3s/registries.yaml
    tmpfs:
    - /run
    - /var/run
    privileged: true
    environment:
    - K3S_URL=https://server:6443
    - K3S_CLUSTER_SECRET=somethingtotallyrandom
    ports:
      - 31000-32000:31000-32000

volumes:
  k3s-server: {}

My registry.yaml file looks as follows

mirrors:
  docker.io:
    endpoint:
      - "http://192.168.2.110:5055"

my private insecure docker registry is defined as

version: '3'
services:
   registry:
     image: registry:2
     ports:
     - 192.168.2.110:5055:5000

Describe the bug

The problem is that even though the registry.yaml is in accordance to the manual I can not pull docker images from a private insecure registry,

To Reproduce

Start the registry and the k3s cluster with docker-compose up

Execute e.g.

docker exec -it $(docker ps |grep "k3s agent" |awk -F\  '{print $1}') crictl pull 192.168.2.110:5055/bla
FATA[2020-05-16T08:47:46.093261937Z] pulling image failed: rpc error: code = Unknown desc = failed to resolve image "192.168.2.110:5055/bla:latest": no available registry endpoint: failed to do request: Head https://192.168.2.110:5055/v2/bla/manifests/latest: http: server gave HTTP response to HTTPS client

(reproducible independent wether image bla exists

Expected behavior

the image is pulled Actual behavior

FATA[2020-05-16T08:47:46.093261937Z] pulling image failed: rpc error: code = Unknown desc = failed to resolve image "192.168.2.110:5055/bla:latest": no available registry endpoint: failed to do request: Head https://192.168.2.110:5055/v2/bla/manifests/latest: http: server gave HTTP response to HTTPS client Additional context / logs

docker exec -it $(docker ps |grep "k3s agent" |awk -F\ '{print $1}') ctr image pull --plain-http 192.168.2.110:5055/bla works like charm (to be reproducible image needs to be pushed to registry first)

[dweomer]

@Drtuna private registries via /etc/rancher/k3s/registries.yaml was not added until 1.0 (see https://github.com/rancher/k3s/issues/99). Have you tried using an up-to-date version of k3s?

[Drtuna]

@Drtuna private registries via /etc/rancher/k3s/registries.yaml was not added until 1.0 (see #99). Have you tried using an up-to-date version of k3s?

Hi, thanks for the feedback I updated accordingly however issue remains k3s version v1.18.2+k3s1 (698e444a)

[wtthetop]

@Drtuna try to edit the config file of docker : /etc/docker/daemon.json, and append "insecure-registries":["IP:PORT"]

[Drtuna]

@Drtuna try to edit the config file of docker : /etc/docker/daemon.json, and append "insecure-registries":["IP:PORT"]

Hi, I added this file (/etc/docker didn't exists) and it didn't help. I also would be surprised as I'm not using docker but crictl. Also the official way seems to be using registries. yaml

[brandond]

In your registry.yaml you've configured your registry as a mirror for docker.io, but you're specifying to pull from it directly - which it expects to be able to do via https. If you want to explicitly pull images from your registry then add a block for it and define the endpoint with http scheme so that it knows not to use https.

[Drtuna]

@brandond thanks for your reply. it helped a lot. correct configuration is

mirrors:
  "192.168.2.110:5055":
    endpoint:
      - "http://192.168.2.110:5055"

is it correct to configure this in the mirrors section?

[brandond]

Upstream documentation is here: https://github.com/containerd/cri/blob/master/docs/registry.md

[davidnuzik]

Using this issue for updating documentation. Documentation could use some more examples.

[BokyLiu]

Hi, there. I'm using k3s an Nvidia Jetson Tx2 and k3s version is v1.18.2+k3s1. I'm meeting the same problem. But in my master tx2, there has no file named registry.yaml but k3s.yaml in /etc/rancher/k3s. So, how can i make currect configuration?

[Drtuna]

Hi, there. I'm using k3s an Nvidia Jetson Tx2 and k3s version is v1.18.2+k3s1. I'm meeting the same problem. But in my master tx2, there has no file named registry.yaml but k3s.yaml in /etc/rancher/k3s. So, how can i make currect configuration?

I just created that file

[onedr0p]

I am also having this issue, I have laid out my information here

[webees]

Similar help is needed.

https://github.com/containerd/containerd/issues/4452

[ttt43ttt]

How can I make it take into effect after updating the private registries in /etc/rancher/k3s/registries.yaml? It seems the crictl pull does not work after changing this file.

[brandond]

@ttt43ttt can you open a new issue describing what you've configured, and what errors you're seeing when trying to pull?

[ttt43ttt]

@ttt43ttt can you open a new issue describing what you've configured, and what errors you're seeing when trying to pull?

https://github.com/rancher/k3s/issues/2156 created

[daniyalrayn]

@brandond thanks for your reply. it helped a lot. correct configuration is

mirrors:
  "192.168.2.110:5055":
    endpoint:
      - "http://192.168.2.110:5055"

is it correct to configure this in the mirrors section?

Yes this is the correct config.

[davidnuzik]

@catherineluse can we get this on your list of Next Ups? The goal of this issue, really, is to see how we can improve our documentation in this area based on the original reported issue plus comments throughout. @brandond is a good contact should you have technical questions.

[Haffon]

I've done following "https://rancher.com/docs/k3s/latest/en/installation/private-registry/" but it dones't work with v1.19.3+k3s2. Should I direct to "https://github.com/containerd/cri/blob/master/docs/registry.md"?

[jonstelly]

I had to rebuild my cluster and I'm stuck on this now. I'm 99% sure this is the same registries.yaml file I had used before.

I've verified that the hostname resolves correctly from my k3s nodes. I know the upstream containerd/kubernetes logging in this area isn't great, but these logs don't seem to tell me anything. Any thoughts or ideas?

my registries.yaml:

mirrors:
  "my.private.registry":
    endpoint:
      - "https://my.private.registry/"
configs:
  "my.private.registry":
    auth:
      username: "me"
      password: "shhhhhhhhh"
    tls:
      ca_file: /usr/local/share/ca-certificates/my-ca.crt

from syslog:

Nov 19 11:52:30 mynode k3s[572577]: E1119 11:52:30.802019  572577 remote_image.go:113] PullImage "my.private.registry/some-image:0.1.0-net50.209" from image service failed: rpc error: code = NotFound desc = failed to pull and unpack image "my.private.registry/some-image:0.1.0-net50.209": failed to resolve reference "my.private.registry/some-image:0.1.0-net50.209": my.private.registry/some-image:0.1.0-net50.209: not found
Nov 19 11:52:30 mynode k3s[572577]: E1119 11:52:30.802138  572577 kuberuntime_manager.go:807] container start failed: ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image "my.private.registry/some-image:0.1.0-net50.209": failed to resolve reference "my.private.registry/some-image:0.1.0-net50.209": my.private.registry/some-image:0.1.0-net50.209: not found
Nov 19 11:52:30 mynode k3s[572577]: E1119 11:52:30.802169  572577 pod_workers.go:191] Error syncing pod 6e410a30-844d-4ac6-87ff-fec69fbf750d ("build-agent-some-image-5c6d44bd49-cg2ds_default(6e410a30-844d-4ac6-87ff-fec69fbf750d)"), skipping: failed to "StartContainer" for "some-image" with ErrImagePull: "rpc error: code = NotFound desc = failed to pull and unpack image \"my.private.registry/some-image:0.1.0-net50.209\": failed to resolve reference \"my.private.registry/some-image:0.1.0-net50.209\": my.private.registry/some-image:0.1.0-net50.209: not found"

[jonstelly]

I'm not exactly sure which caused the issue, but I got it working again. My issue was either the quotation marks or the trailing slash on the endpoint URL. This works for me:

mirrors:
  my.private.registry:
    endpoint:
      - https://my.private.registry
configs:
  my.private.registry:
    auth:
      username: me
      password: shhhhhh
    tls:
      ca_file: /usr/local/share/ca-certificates/my-ca.crt

[Haffon]

Hi, everyone, I have 2 issues and want to get some help: 1.Firstly I deployed 1 k3s server and 1 k3s agent, I switch to docker engine in the server since I can't fix the private registry issue. then the server use the images which already pull and load from private registry alone side k3s command. There is no error "failed to get sandbox image "docker.io/rancher/pause:3.1" now. but the agent side still popup this error message whether run with --docker or not. I want to switch back to containerd engine in server side, but I still got "failed to get sandbox image "docker.io/rancher/pause:3.1" error message. 2. I continuesly got "TLS handshake error from X.Y.Z.M:nnn: remote error: tls: bad certificate" error message, does the server accept the agent or not? how do I put TLS files on, or disable the TLS options? @jonstelly Is there no quotation marks with endpoint URL? I just created and edit registries.yaml file, and put it in /etc/rancher/k3s/ directory, and it does not take effect. here is the file content:

mirrors:
  A.B.C.D:
    endpoint:
      - "http://A.B.C.D:5000"
configs:
  "A.B.C.D:5000":
    auth:
      username: user # this is the registry username
      password: pass # this is the registry password

note: the username and password is omit for security reason.

[jonstelly]

I bet you just need to add the tls/certificate settings like mentioned here. For me, I generated my registry certificate and signed it with my CA certificate so I only used ca_file and that seems to work. But if you did a self-signed certificate you may just need to specify cert_file.

[Haffon]

I think I missed the order of question and intro in my previous comment. I am using k3s 1.18.10-k3s1 and just have 2 questions:

1. k3s server or agent side doesn't login and pull image from private registry. This private registry using HTTP basic authentication, so there is only username and password, no need TLS ca_file/cert_file. so the registries.yaml file is setted as my previous comment. I think it is acceptable if we update our private registry to use HTTP digest authentication or TLS.
2. The k3s server print "remote error: tls: bad certificate", even we can build a private CA root, we don't know how to deployment our ca_file/cert_files to all relatied nodes. Is there any document to show how to build a k3s cluster upon TLS, with the operation details about how to build CA department and sign a cert_file for each node, and the k3s server and agent know where to read the file to connect each other with TLS?

[brandond]

If you're using private PKI (your own self-signed certs, or certs signed with a private CA) you will need to handle distribution of those files manually, along with configuration of whatever software needs to trust those certs.

[Haffon]

Hi @brandond , Does k3s use system wide authentication logic according to "remote error: tls: bad certificate" message? Does the software need to trust those certs is distributed alone side k3s but with Linux OS?

[brandond]

For purposes of authenticating remote registries, and absent any specific configuration, containerd will use the behavior documented here: https://golang.org/src/crypto/x509/root_linux.go

[Haffon]

Thanks all of you, I think I need make a clean enviroment to deployment k3s again, then I will try post issue details here.

[tinyzimmer]

I'm not exactly sure which caused the issue, but I got it working again. My issue was either the quotation marks or the trailing slash on the endpoint URL. This works for me:

mirrors:
  my.private.registry:
    endpoint:
      - https://my.private.registry
configs:
  my.private.registry:
    auth:
      username: me
      password: shhhhhh
    tls:
      ca_file: /usr/local/share/ca-certificates/my-ca.crt

Holy heck, this was exactly the issue I just ran into and beat my head on for 4 hours. No quotes and the no trailing slashes was the fix. It was weird because containerd seemed to be parsing the generated config just fine and was rather quiet about it's issues with it.

[boindil]

This hast recently stopped working for me. I've tried with and without quotes. It's worth noting, that I didnt change anything except for upgrading k3s.

[abdennour]

in the docs, it said :

In order for the registry changes to take effect, you need to restart K3s on each node.

What should i run and where ? systemctl restart k3s or k3s command ?

UPDATE

in master, restarting by systemctl restart k3s... in node, restarting by systemctl restart k3s-agent