Cannot use `--container-runtime-endpoint` with rootless k3s

[brandond]

from @hinshun

Due to the rootlesskit setup being embedded in the k3s binary, it's impossible to use k3s server --rootless with an external -container-runtime-endpoint

1. kubelet (in k3s) fail to startup due to health check of containerd-runtime-endpoint
2. containerd (outside of k3s) fail to startup due to rootlesskit not yet self-exec'd (i.e. child_pid not yet available).

[brandond]

@hinshun if you could document how you are running containerd rootless outside of k3s, that would be helpful in closing out this issue.

[hinshun]

I am providing a NixOS module that setups a systemd user service with this as the entrypoint: https://github.com/pdtpartners/nix-snapshotter/blob/main/modules/common/containerd-rootless.sh

Essentially it just runs rootlesskit:

rootlesskit \
    --net=slirp4netns \
    --disable-host-loopback \
    --copy-up=/etc \
    --copy-up=/run \
    --copy-up=/var/lib \
    --port-driver=slirp4netns \
    --state-dir="${XDG_RUNTIME_DIR}/containerd-rootless" \
    sh -c "containerd-rootless-child @containerdArgs@"

Inside containerd-rootless-child, we do essentially the same thing as k3s bind mount setup except in bash: https://github.com/pdtpartners/nix-snapshotter/blob/main/modules/common/containerd-rootless-child.sh

for i in "${!mountSources[@]}"; do
  mountSource=${mountSources[$i]}
  mountPoint=${mountPoints[$i]}

  # Remove the *symlinks* for the existing files in the parent namespace if any,
  # so that we can create our own files in our mount namespace.
  # The actual files in the parent namespace are *not removed* by this rm command.
  rm -rf "$mountPoint"

  echo >&2 Bind mounting ${mountSource} to ${mountPoint} inside mount namespace
  mkdir -p "$mountSource" "$mountPoint"
  mount --bind "$mountSource" "$mountPoint"
done

[brandond]

Do you have any way to do this on a more traditional distro? We usually test on SLE, Ubuntu, and EL. NixOS is not on our support matrix and we're not particularly familiar with how to test on it.

[hinshun]

To be clear, #9309 does not fix this issue, it only exposes the containerd socket via $XDG_RUNTIME_DIR/k3s/containerd/containerd.sock when running k3s in rootless mode. For the purposes of cherry-picking #9309 for release, the verification step is something like:

k3s server --rootless
ctr -a "$XDG_RUNTIME_DIR/k3s/containerd/containerd.sock" -n k8s.io image ls

Verifying that after #9309 it is still impossible to use --container-runtime-endpoint is difficult as its a matter of understanding that there's a startup deadlock between a rootless containerd & rootless k3s.

[rancher-max]

@hinshun it sounds like more work is needed in that case to get this fully working? I'm inclined to leave this issue open but bump the milestone out to a future release for a more complete resolution. Thank you for the PR and work on this! I'm glad that there is something usable at this point for Feb releases and look forward to an even stronger feature in the future 🎉 👏