k3d [FEATURE] Handle known issues with some storage backends (btrfs, zfs, ...)

In v5.0.0 we should try to handle some known issues with storage backends more conveniently. We already have FAQ entries for btrfs and zfs. At the very least we should check for the storage backend (we already have the runtime-info command) and based on that information emit a warning log.

[ ] automatically add /dev/mapper:/dev/mapper mount on btrfs
- [ ] check, if, by now, this could work for zfs as well (though k3s may still be lacking zfs support as mentioned also in the FAQ entry)
  - [ ] if not, emit a warning with a hint to the FAQ entry on ZFS systems

References

how kind detects whether this is needed or not: https://github.com/kubernetes-sigs/kind/blob/b6bc112522651d98c81823df56b7afa511459a3b/pkg/cluster/internal/providers/docker/util.go#L51-L87

Jun 11 '21 11:06 iwilltry42

As a pointer for others, these steps allowed me to run K3d in unpriviledged Podman containers (ulimit increased, CPU and CPUSET delegation enabled) on Ubuntu ZFS:

Inspiration

https://www.enricobassetti.it/2022/02/k3s-zfs-cgroups-v2/
https://github.com/k3s-io/k3s/issues/1688#issuecomment-619570374

sudo zfs create -o mountpoint=/srv/jobs pool/jobs
sudo zfs create pool/jobs/k3d
sudo chown 1000:1000 /srv/jobs/k3d
touch /srv/jobs/k3d/k3d-config.yaml

sudo zfs create -s -V 10GB pool/jobs/k3d/overlay-server0
sudo zfs create -s -V 10GB pool/jobs/k3d/overlay-server1
sudo zfs create -s -V 10GB pool/jobs/k3d/overlay-server2
sudo zfs create -s -V 10GB pool/jobs/k3d/overlay-agent0
sudo zfs create -s -V 10GB pool/jobs/k3d/overlay-agent1
sudo zfs create -s -V 10GB pool/jobs/k3d/overlay-agent3
sudo mkfs.ext4 /dev/zvol/pool/jobs/k3d/overlay-server0
sudo mkfs.ext4 /dev/zvol/pool/jobs/k3d/overlay-server1
sudo mkfs.ext4 /dev/zvol/pool/jobs/k3d/overlay-server2
sudo mkfs.ext4 /dev/zvol/pool/jobs/k3d/overlay-agent0
sudo mkfs.ext4 /dev/zvol/pool/jobs/k3d/overlay-agent1
sudo mkfs.ext4 /dev/zvol/pool/jobs/k3d/overlay-agent2
sudo mount /dev/zvol/pool/jobs/k3d/overlay-server0 /srv/jobs/k3d/overlay-server0
sudo mount /dev/zvol/pool/jobs/k3d/overlay-server1 /srv/jobs/k3d/overlay-server1
sudo mount /dev/zvol/pool/jobs/k3d/overlay-server2 /srv/jobs/k3d/overlay-server2
sudo mount /dev/zvol/pool/jobs/k3d/overlay-agent0 /srv/jobs/k3d/overlay-agent0
sudo mount /dev/zvol/pool/jobs/k3d/overlay-agent1 /srv/jobs/k3d/overlay-agent1
sudo mount /dev/zvol/pool/jobs/k3d/overlay-agent2 /srv/jobs/k3d/overlay-agent2

These steps helped me debugging ¹:

podman unshare ls -land /srv/jobs/k3d/overlay-*/
ls -land /srv/jobs/k3d/overlay*/
podman unshare chown -R 0:0 /srv/jobs/k3d/overlay-*/
# and when that didn't work
sudo chown -R 1000:1000 /srv/jobs/k3d/overlay-*/
sudo rm /srv/jobs/k3d/overlay-*/lost+found

This configuration was used in a cluster's k3d-config.yaml (the :z helps with SELinux and Podman):

---
apiVersion: k3d.io/v1alpha4
…
servers: 3
agents: 3
…

volumes:
  - volume: "/srv/jobs/k3d/overlay-server0:/var/lib/rancher/k3s/agent/containerd:z"
    nodeFilters:
      - server:0
  - volume: "/srv/jobs/k3d/overlay-server1:/var/lib/rancher/k3s/agent/containerd:z"
    nodeFilters:
      - server:1
  - volume: "/srv/jobs/k3d/overlay-server2:/var/lib/rancher/k3s/agent/containerd:z"
    nodeFilters:
      - server:2
  - volume: "/srv/jobs/k3d/overlay-agent0:/var/lib/rancher/k3s/agent/containerd:z"
    nodeFilters:
      - agent:0
  - volume: "/srv/jobs/k3d/overlay-agent1:/var/lib/rancher/k3s/agent/containerd:z"
    nodeFilters:
      - agent:1
  - volume: "/srv/jobs/k3d/overlay-agent2:/var/lib/rancher/k3s/agent/containerd:z"
    nodeFilters:
      - agent:2

…

options:
  …
  k3s:
    extraArgs:
      - arg: "--kubelet-arg=feature-gates=KubeletInUserNamespace=true"
        nodeFilters:
        - "all"
      …

With these adaptions, it was possible to run K3d via rootless Podman on ZFS as an unpriviledged user:

export DOCKER_SOCK=$XDG_RUNTIME_DIR/podman/podman.sock
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
podman network create cluster0
k3d registry create --default-network cluster0 -p 5000 registry
K3D_FIX_CGROUPV2=true k3d cluster create --config /srv/jobs/k3d/k3d-config.yaml

This is all still a bit verbose, and could be simplified with using fstab for the outer mount of the ext4 filesystems, and could be scripted easily.

And with IPv6

For IPv6 support, replace the podman network create cluster0 line with:

cp /usr/share/containers/containers.conf ~/.config/
sed -i 's/#network_backend = ""/network_backend = "netavark"' ~/.config/containers.conf
podman system reset # deletes all
podman info | rg -i networkbackend
podman network create --ipv6 cluster0

For network configuration, I have followed the example in the upstream documentation about enabling IPv6 ¹ ²

…
options:
…
  k3s:
    extraArgs:
      …
      - arg: "--kubelet-arg=node-ip=::"
        nodeFilters:
        - "all"
      - arg: "--cluster-cidr=10.244.0.0/16,2001:db8:42:0::/56"
        nodeFilters:
        - "server:*"
      - arg: "--service-cidr=10.96.0.0/16,2001:db8:42:1::/112"
        nodeFilters:
        - "server:*"
      …

As a bonus, in this expressive way we get nice filesystem usage statistics per node:

$ zfs list -r -t volume pool/jobs/k3d
NAME                                             USED  AVAIL     REFER  MOUNTPOINT
pool/jobs/k3d/overlay-server0         558M  1.16T      558M  -
pool/jobs/k3d/overlay-server1         158M  1.16T      158M  -
pool/jobs/k3d/overlay-server2         158M  1.16T      158M  -
pool/jobs/k3d/overlay-agent0         21.2M  1.16T     21.2M  -
pool/jobs/k3d/overlay-agent1         21.4M  1.16T     21.4M  -
pool/jobs/k3d/overlay-agent2         21.8M  1.16T     21.8M  -

If we're running above 10 GiB for each here, we can use ZFS and ext4 to increase the volumes.

I am pretty sure this workflow also applies partly to K3s, especially where the ZVOLs and ext4 filesystem mounts are concerned:

https://github.com/k3s-io/k3s/discussions/3980

Also this documentation page appears to be a little outdated, in terms of what one could do, and should probably point here instead:

https://k3d.io/v5.4.6/faq/faq/#issues-with-zfs

Please also note that ZFS have recently merged commits, which produce compatibility with OverlayFS and idmapped mounts, therefore containers in general, so this shouldn't remain a problem in the near future:

https://github.com/openzfs/zfs/commit/e015d6cc0b60d4675c9b6d2433eed2c8ef0863e8
https://github.com/openzfs/zfs/pull/9549
https://github.com/openzfs/zfs/pull/13671

Improving native support for containers on ZFS is one of the current main priorities for the next OpenZFS releases, so your feedback would be highly appreciated.

Dec 02 '22 00:12 almereyda

Please note that ZFS 2.2.0 with Linux Container Support, such as OverlayFS, is nearing completion:

https://github.com/openzfs/zfs/releases?q=2.2.0&expanded=true

Jul 29 '23 21:07 almereyda

k3d k3d copied to clipboard

[FEATURE] Handle known issues with some storage backends (btrfs, zfs, ...)

References

k3d
k3d copied to clipboard