k0sctl apply fails on Alpine Linux becuase cgroups service is not running

[Ikke]

When running k0sctl apply targeting an Alpine Linux worker, the worker fails to start due to the cgroups service is not running.

k0s sysinfo reports:

Control Groups: unexpected file system type of "/sys/fs/cgroup": 0x62656572 (rejected)
  cgroup controller "cpu": unexpected file system type of "/sys/fs/cgroup": 0x62656572 (rejected)
  cgroup controller "cpuacct": unexpected file system type of "/sys/fs/cgroup": 0x62656572 (rejected)
  cgroup controller "cpuset": unexpected file system type of "/sys/fs/cgroup": 0x62656572 (rejected)
  cgroup controller "memory": unexpected file system type of "/sys/fs/cgroup": 0x62656572 (rejected)
  cgroup controller "devices": unexpected file system type of "/sys/fs/cgroup": 0x62656572 (rejected)
  cgroup controller "freezer": unexpected file system type of "/sys/fs/cgroup": 0x62656572 (rejected)
  cgroup controller "pids": unexpected file system type of "/sys/fs/cgroup": 0x62656572 (rejected)
  cgroup controller "hugetlb": unexpected file system type of "/sys/fs/cgroup": 0x62656572 (rejected)
  cgroup controller "blkio": unexpected file system type of "/sys/fs/cgroup": 0x62656572 (rejected)

After starting the service (and making sure it's enabled at boot), this issue is fixed, but I had to reset k0s before I could apply again:

# rc-service cgroups start
# rc-update add cgroups

• Alpine Linux version: 3.19
• k0sctl version: v0.16.0

[twz123]

Just to make sure I understand this correctly: The problem is that a reapply via k0sctl required a k0s reset? Maybe a k0sctl apply --force would have done the trick as well. We could also think about running k0s sysinfo before installing k0s, so that we can circumvent this problem.

[twz123]

0x62656572 is sysfs, which is probably sort of the expected outcome if there's no dedicated cgroupfs mounted, so we could have a nicer error message, along with a link to the documentation.

[Ikke]

After enabling cgroup, the worker failed to start with:

Error: failed to decode join token: illegal base64 data at input byte 0

k0sctl apply failed with:

INFO * Running clean-up for phase: Acquire exclusive host lock
FATA apply failed - log file saved to /root/.cache/k0sctl/k0sctl.log: failed on 1 hosts:
 - [ssh] 10.254.0.3:22: k0s doesn't appear to be running but has been installed as a service at /etc/init.d/k0sworker - please remove it or start the service

I did not try k0sctl apply --force though.

[twz123]

Tackling the easiest part first:

After enabling cgroup, the worker failed to start with:

Error: failed to decode join token: illegal base64 data at input byte 0

Would that message be better?

Error: token file "/path/to/file" is empty: obtain a new token via "k0s token create ..." and store it in the file or reinstall this node via "k0s install --force ..." or "k0sctl apply --force ..." ( k0sproject/k0s#4341)

[Ikke]

Would that message be better?

Yes, that would be definitely better. The suggested error tells you what token (file) is the issue, why it's invalid and even how to correct it.

[Ikke]

Ultimately it would be nice if k0sctl would enable these services automatically, similar how it installs some packages.

[ncopa]

I think we should fix this in k0s.