kind [zfs] kind create cluster using podman provider fails to bootstrap the cluster

What happened:

Creating a cluster with kind create cluster under rootful and rootless podman hangs on ⢎⡠ Starting control-plane 🕹️ and later fails with

[kubelet-check] Initial timeout of 40s passed. 114 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s in 2 milliseconds

~ » sudo kind create cluster                                                                                                                                                                        
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.24.0) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
⢎⡠ Starting control-plane 🕹️ ^C

What you expected to happen:

I'd expect the kind create cluster to successfully create a cluster.

How to reproduce it (as minimally and precisely as possible):

On a zfs backed Arch system installation running podman (rootful or rootless) with fuse-overlayfs storage driver execute kind create cluster.

Anything else we need to know?:

Running kind with --retain flag and peeking inside the container we see:

# journalctl -u kubelet.service
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.457606     223 kubelet.go:2419] "Error getting node" err="node \"kind-control-plane\" not found"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.524655     223 dns.go:157] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: fc00:f853:ccd:e793::1 10.89.0.1 84.208.20.110"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.558482     223 kubelet.go:2419] "Error getting node" err="node \"kind-control-plane\" not found"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.637581     223 remote_runtime.go:201] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: failed to mount rootfs component &{overlay overlay [index=off workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/work upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/fs lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs]}: invalid argument: unknown"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.637641     223 kuberuntime_sandbox.go:70] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: failed to mount rootfs component &{overlay overlay [index=off workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/work upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/fs lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs]}: invalid argument: unknown" pod="kube-system/etcd-kind-control-plane"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.637675     223 kuberuntime_manager.go:815] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: failed to mount rootfs component &{overlay overlay [index=off workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/work upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/fs lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs]}: invalid argument: unknown" pod="kube-system/etcd-kind-control-plane"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.637744     223 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"etcd-kind-control-plane_kube-system(f3dbec4949f2648b73c0b4e85ed47e2c)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"etcd-kind-control-plane_kube-system(f3dbec4949f2648b73c0b4e85ed47e2c)\\\": rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: failed to mount rootfs component &{overlay overlay [index=off workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/work upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/fs lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs]}: invalid argument: unknown\"" pod="kube-system/etcd-kind-control-plane" podUID=f3dbec4949f2648b73c0b4e85ed47e2c
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.658726     223 kubelet.go:2419] "Error getting node" err="node \"kind-control-plane\" not found"
Aug 08 19:47:12 kind-control-plane kubelet[223]: E0808 19:47:12.758958     223 kubelet.go:2419] "Error getting node" err="node \"kind-control-plane\" not found"

All logs: logs.tar.gz

The same happens under rootless podman.

Environment:

kind version: (use kind version): kind v0.14.0 go1.18.3 linux/amd64
Kubernetes version: (use kubectl version): node image is 1.24.0
Docker version: (use docker info):

~ » sudo podman info                                                                                                                                                                           1 ↵ 
host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.3-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.3, commit: ab52a597278b20173440140cd810dc9fa8785c93'
  cpuUtilization:
    idlePercent: 92.57
    systemPercent: 3.14
    userPercent: 4.28
  cpus: 8
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: <redacted>
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.18.16-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 16413462528
  memTotal: 24849227776
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.5-1
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 24m 20.01s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 381226975232
  graphRootUsed: 34348859392
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.1.1
  Built: 1659559968
  BuiltTime: Wed Aug  3 22:52:48 2022
  GitCommit: f73d8f8875c2be7cd2049094c29aff90b1150241-dirty
  GoVersion: go1.19
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.1

OS (e.g. from /etc/os-release):

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux-logo

Aug 08 '22 19:08 maciekmm

Are those logs from the rootful or rootless logs?

Is rootful podman running with fuse-overlayfs?

I don't have an environment with zfs x podman, but when we detect zfs inside the kind node we have to switch to the "native" driver, with lower precedence than on rootless (userns remapping) we will need to use fuse-overlayfs / overlayfs depending on the host kernel.

We seem to be runing with overlayfs snapshotter inside the container, it probably needs to be fuse-overlayfs or native.

you can override this with KIND_EXPERIMENTAL_SNAPSHOTTER=native kind create cluster (note: native may have poor performance)

https://github.com/kubernetes-sigs/kind/blob/3ad5fe9ee9c8e5071ac03aff87ac3907c103dd42/images/base/files/usr/local/bin/entrypoint#L102-L128

Aug 08 '22 20:08 BenTheElder

The attached logs were from rootful podman. With or without mount_program set to fuse-overlayfs (I removed it while debugging) the behavior is the same.

Rootful

Running sudo kind create cluster fails :red_square: Running sudo KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=fuse-overlayfs kind create cluster works :heavy_check_mark: Running sudo KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=native kind create cluster works :heavy_check_mark:

Running stat -f -c %T /kind inside the container returns fuseblk

Rootless

Running kind create cluster fails :red_square: Running KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=fuse-overlayfs kind create cluster works :heavy_check_mark: Running KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=native kind create cluster works :heavy_check_mark:

So it seems like we are picking the wrong snapshotter. I tried running some of the commands from overlayfs_prefferable and they seem to succeed under the overlayfs snapshotter:

root@kind-control-plane:/#   tmp=$(mktemp -d)
root@kind-control-plane:/#   mkdir -p "${tmp}/l" "${tmp}/u" "${tmp}/w" "${tmp}/m"
root@kind-control-plane:/# $?
bash: 0: command not found
root@kind-control-plane:/# mount -t overlay -o lowerdir="${tmp}/l,upperdir=${tmp}/u,workdir=${tmp}/w" overlay "${tmp}/m"
root@kind-control-plane:/# $?
bash: 0: command not found

It seems like it tries to use the overlayfs but then the real invocation of it fails with invalid argument. Not sure how useful that is as I lack some knowledge here.

Thanks for the pointers @BenTheElder . Happy to help with this further.

Aug 08 '22 21:08 maciekmm

Thanks!

I think we should probably rework the snapshotter selection to be independent of the rest of userns config and always select fuse-overlays when we’re in fuseblk. It seems like a safe bet that we’re in fuse-overlayfs if /kind is on fuseblk

Aug 09 '22 10:08 BenTheElder

I also encountered fails this morning when running both with podman and docker; I am also on Arch

Aug 10 '22 11:08 caniko

@maciekmm could you make a PR for the docs where you provide a guide on how to run kind with rootless podman on Arch? At least explain it here.

Command:

KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=fuse-overlayfs kind create cluster

Error:

ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

Aug 10 '22 11:08 caniko

working on a fix here https://github.com/kubernetes-sigs/kind/pull/2874

Aug 10 '22 17:08 BenTheElder

test image docker.io/bentheelder/kind-node:v1.24.3@sha256:9fdef952a166325f89cae70b7e69767b50b7a7b0ace1f03df362228b82ee34e4 contains the proposed image fix from #2874

use with kind create cluster --image=... (should be compatible with current stable KIND)

Aug 10 '22 18:08 BenTheElder

@BenTheElder thanks for working on this.

It seems like the image you have linked does not contain the fixes.

sudo podman ps
CONTAINER ID  IMAGE                                                                                                    COMMAND     CREATED        STATUS            PORTS                      NAMES
6d883a2f508d  docker.io/bentheelder/kind-node@sha256:9fdef952a166325f89cae70b7e69767b50b7a7b0ace1f03df362228b82ee34e4              5 minutes ago  Up 4 minutes ago  127.0.0.1:41301->6443/tcp  kind-control-plane

~ » sudo podman exec -it 6d883a2f508d grep -A30 SNAPSHOTTER /usr/local/bin/entrypoint                                                                                                              
    echo "WARN: UserNS: SELinux might be Enforcing. If you see an error related to overlayfs, try setting \`KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=fuse-overlayfs\` ." >&2
  fi
  return 0
}

configure_containerd() {
  local snapshotter=${KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER:-}
  if [[ -n "$userns" ]]; then
    # userns (rootless) configs

    # Adjust oomScoreAdj
    sed -i 's/restrict_oom_score_adj = false/restrict_oom_score_adj = true/' /etc/containerd/config.toml

    # Use fuse-overlayfs if overlayfs is not preferrable: https://github.com/kubernetes-sigs/kind/issues/2275
    if [[ -z "$snapshotter" ]] && ! overlayfs_preferrable; then
      snapshotter="fuse-overlayfs"
    fi
  else
    # we need to switch to the 'native' snapshotter on zfs
    if [[ -z "$snapshotter" ]] && [[ "$(stat -f -c %T /kind)" == 'zfs' ]]; then
      snapshotter="native"
    fi
  fi
  if [[ -n "$snapshotter" ]]; then
    echo "INFO: changing snapshotter from \"overlayfs\" to \"$snapshotter\""
    sed -i "s/snapshotter = \"overlayfs\"/snapshotter = \"$snapshotter\"/" /etc/containerd/config.toml
    if [[ "$snapshotter" = "fuse-overlayfs" ]]; then
      echo 'INFO: enabling containerd-fuse-overlayfs service'
      systemctl enable containerd-fuse-overlayfs
    fi
  fi
}

configure_proxy() {
  # ensure all processes receive the proxy settings by default
  # https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html
  mkdir -p /etc/systemd/system.conf.d/

Aug 10 '22 19:08 maciekmm

Thanks, let me see if I grabbed the wrong image or built wrong and update.

Aug 10 '22 20:08 BenTheElder

fixed, should be docker.io/bentheelder/kind-node:v1.24.3@sha256:4b649f5099d392a97f8031d4ff0730c2a13392fc18879b08a982768d00686ec1

Aug 10 '22 20:08 BenTheElder

Thanks! Can confirm this works under both rootful and rootless podman on zfs running fuse overlayfs. :tada:

Aug 10 '22 20:08 maciekmm

Any snapshots for v1.22?

Aug 11 '22 07:08 caniko

Thanks for confirming @maciekmm 🙏

@caniko No, but you can build your own if you checkout the PR https://kind.sigs.k8s.io/docs/user/quick-start/#building-images

Aug 11 '22 17:08 BenTheElder

should be fixed by https://github.com/kubernetes-sigs/kind/pull/2874

current plan is to try to cut a release next week alongside kubernetes 1.25

Aug 18 '22 21:08 BenTheElder

kind
kind copied to clipboard

[zfs] kind create cluster using podman provider fails to bootstrap the cluster - failed to mount rootfs component

Rootful

Rootless

kind kind copied to clipboard

[zfs] kind create cluster using podman provider fails to bootstrap the cluster - failed to mount rootfs component

Rootful

Rootless

kind
kind copied to clipboard