k3s server service exits on VMWare VM running CentOS 7

[divick]

Environmental Info: K3s Version: k3s version v1.24.3+k3s1 (990ba0e8) go version go1.18.1

 dmidecode | grep Product
Product Name: VMware Virtual Platform
Product Name: 440BX Desktop Reference Platform

Node(s) CPU architecture, OS, and Version: Linux colmbapp208133 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug: I am trying to install and run k3s on a VMWare virtual machine with "native snapshotter" with:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--snapshotter native" sh -s -

I did this because earlier even though k3s service was able to start but it was showing following warning:

 server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": /var/lib/rancher/k3s/agent/containerd does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"

Now I don't get the above warning but I get the following warnings and an error:

sh: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
sh: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
modprobe: modprobe: FATAL: Module br_netfilter not found.
...
...
 k3s: time="2022-08-12T18:17:31+05:30" level=error msg="Failed to set sysctl: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory"
k3s: time="2022-08-12T18:17:31+05:30" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
k3s: time="2022-08-12T18:17:31+05:30" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
k3s: W0812 18:17:31.698319   21188 genericapiserver.go:557] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources.
k3s: I0812 18:17:31.699756   21188 instance.go:274] Using reconciler: lease
k3s: containerd: exit status 1
systemd: k3s.service: main process exited, code=exited, status=1/FAILURE

Steps To Reproduce:

• Installed K3s:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--snapshotter native" sh -s -

Expected behavior: k3s service should have started fine and not exited. I am not sure what should be the system requirements to run k3s and what kind of VMs can it run on.

Actual behavior:

● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2022-08-12 18:28:27 IST; 657ms ago
     Docs: https://k3s.io
  Process: 54431 ExecStart=/usr/local/bin/k3s server --snapshotter native (code=exited, status=1/FAILURE)
  Process: 54428 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 54424 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=1/FAILURE)
  Process: 54420 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
 Main PID: 54431 (code=exited, status=1/FAILURE)

Aug 12 18:28:27 colmbapp208133 systemd[1]: Failed to start Lightweight Kubernetes.
Aug 12 18:28:27 colmbapp208133 systemd[1]: Unit k3s.service entered failed state.
Aug 12 18:28:27 colmbapp208133 systemd[1]: k3s.service failed.

[brandond]

This may be a duplicate of https://github.com/k3s-io/k3s/issues/5912 but the error /var/lib/rancher/k3s/agent/containerd does not support d_type you're getting indicates that your root filesystem is not configured properly to run the overlayfs snapshotter. I would recommend putting /var/lib/rancher on a different partition that either uses ext4, or xfs with the correct options.

[divick]

@brandond As stated above, I have tried with snapshotter=native as well as fuse-overlayfs too but in that case it gives the following message in logs:

 modprobe: FATAL: Module br_netfilter not found

But I see that in my linux kernel it is compiled in rather than as module:

CONFIG_BRIDGE_NETFILTER=y

so anyways if k3s is looking for br_netfilter to be loaded or loads it explicitly, it will fail. So it seems unrelated to #5912

[brandond]

Those modprobe ExecPre entries are prefixed with - which means they are allowed to fail. Something else is going on. Fix your xfs filesystem issue, and see if any other problems remain.

[divick]

@brandond ok. But I have only access to these vms which have these as defaults. It may not be possible for me to update the filesystem option for xfs for the mounted partition on /var.

[stale[bot]]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.