k3s

k3s copied to clipboard

I'm using k3s v0.9.0-rc2 on an array of Raspberry Pi 4 (4 GB) with an unofficial ARM64 image of Ubuntu Server 18.04.3 (got it from here). To set the version, the following assumes this setup:

export INSTALL_K3S_VERSION=v0.9.0-rc2

... but I get the same results with v0.8.1.

When I use the standard installation depending on Containerd, everything is running fine. However, I want to use Docker for some specific features, so I install Docker using

apt-get update
apt-get install containerd.io=1.2.6-3 docker-ce=5:18.09.9~3-0~ubuntu-bionic docker-ce-cli=5:18.09.9~3-0~ubuntu-bionic

Now I install the the master node using this:

curl -fsL https://get.k3s.io | sh -s server --docker

... and the worker nodes using that:

export K3S_TOKEN=[...]
export K3S_URL=[...]
curl -fsL https://get.k3s.io | sh -s agent --docker

So far, so good, everything is working fine. However, the Kubernetes documentation says that the Docker daemon should use the systemd CGroup driver instead of the default cgroupfs, so I'm adding /etc/docker/daemon.json with the following content:

{
  "exec-opts": [
    "native.cgroupdriver=systemd"
  ],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}

... and restart the Docker daemon. Next, I reinstall k3s on the master node using this:

curl -fsL https://get.k3s.io | sh -s server --docker --kubelet-args cgroup-driver=systemd

... and the following for the worker nodes:

export K3S_TOKEN=[...]
export K3S_URL=[...]
curl -fsL https://get.k3s.io | sh -s agent --docker --kubelet-args cgroup-driver=systemd

However, this time the worker nodes are not added to the cluster and when I do kubectl describe pod/coredns-b7464766c-nq9gm --namespace kube-system then I see the following event:

Failed create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "coredns-b7464766c-nq9gm": Error response from daemon: cgroup-parent for systemd cgroup should be a valid slice named as "xxx.slice"

What is wrong?

For comparison, the same Docker configuration works just fine when installing original Kubernetes using kubeadm, so this issue is specific to k3s.

The kubernetes docs just show an example, you can keep to user the cgroupfs driver in docker.

The k3s agent/kubelet is hard code the cgroupfs type now: https://github.com/rancher/k3s/blob/8a9c8675c2a16591e13a655617acdb1e75d3962b/pkg/daemons/agent/agent.go#L65

Ok, I see. Well, I think this issue is a feature request then. The reasoning in the Kubernetes docs is sound and I'ld love to see this supported by k3s, too.

systemd cgroup driver is not supported because systemd will not allow statically linked binaries (which k3s is built on). The cgroups manager code needs something from systemd CGO so we have to disable it.

The systemd cgroup support from k8s/docker in general is quite poor. I'd never recommend the systemd driver and am quite surprised the k8s docs do.

As the world moves to cgroups v2 and systemd continues its quest to ruin everything, the case may be different and the systemd driver may be the best technical choice, but v2 support is still in the works.

When will the government support cgroup-driver=systemd? Will they not support it in the future?

Hello,

as this issue seems not wanted to be resolved, the question is:

What is the recommended configuration for the cgroup-driver when running k3s on an OS with systemd, like on Ubuntu? I was running 3 clusters initiated with kubeadm and docker running, where the cgroup-driver was cgroupfs. This was not a good setup, as two different cgroup-drivers making conflicts...

(This issue IMHO is independent from using docker or not, isn't it?)

@ibuildthecloud I understand that cgroupfs v2 is superb compared to cgroupfs v1. But which runtime (containerd, docker, podman, podman + crun) supports both cgroupfs v2 and k3s on modern but stable LTS OS?

We have supported the systemd cgroup driver for quite a while. It should work fine with docker as long as the kubelet is also manually configured for the same driver.