microk8s
microk8s copied to clipboard
canonical
The memory cgroup is not enabled.
I appended cgroup_enable=memory cgroup_memory=1 to cmdline.txt file in each node and reboot but it not working.
microk8s inspect
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
Building the report tarball
Report tarball is at /var/snap/microk8s/1669/inspection-report-20201027_221452.tar.gz
I think some rpis linux distro is in /boot/firmware/nobtcmd.txt
https://microk8s.io/docs/install-alternatives#heading--arm
I think some rpis linux distro is in
/boot/firmware/nobtcmd.txthttps://microk8s.io/docs/install-alternatives#heading--arm
I added cgroup_enable=memory cgroup_memory=1 to/boot/firmware/nobtcmd.txt as well, but it's didn't work.
ubuntu@ubuntu:/$ cat boot/firmware/nobtcmd.txt
cgroup_enable=memory cgroup_memory=1
microk8s inspect
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
Building the report tarball
Report tarball is at /var/snap/microk8s/1669/inspection-report-20201102_155946.tar.gz
@UsernameAlvarez just to confirm could you attach the inspection tarball?
I would like to know, this is on a real hardware, not any virtual machine on ARM64.
In the API server logs I see:
Nov 02 15:01:25 ubuntu microk8s.daemon-apiserver[1993946]: Error: start node: raft_start(): io: load closed segment 0000000013312357-0000000013312881: found 524 entries (expected 525)
Nov 02 15:01:25 ubuntu systemd[1]: snap.microk8s.daemon-apiserver.service: Main process exited, code=exited, status=1/FAILURE
@freeekanayaka may have more insight on this error.
@UsernameAlvarez could you remove microk8s and reinstall it from the latest edge channel. We have a number of fixes in that channel and we will be releasing them soon.
sudo snap remove microk8s --purge
sudo snap install microk8s --classic --channel=1.19/edge
@ktsakalozos Thanks, I did it but is not working.
microk8s (1.19/edge) v1.19.3 from Canonical✓ installed
microk8s inspect
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
Building the report tarball
Report tarball is at /var/snap/microk8s/1783/inspection-report-20201104_230237.tar.gz
Inspection-report inspection-report-20201104_230237.tar.gz
@UsernameAlvarez can you cat the file /boot/firmware/cmdline.txt. And cat /proc/cgroups
@balchua yes.
$ cat /boot/firmware/cmdline.txt
net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc
cgroup_enable=memory cgroup_memory=1
$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 5 1 1
cpu 8 1 1
cpuacct 8 1 1
blkio 3 1 1
memory 0 113 0
devices 9 104 1
freezer 10 2 1
net_cls 2 1 1
perf_event 6 1 1
net_prio 2 1 1
pids 4 111 1
rdma 7 1 1
I also have this problem, and have applied the 1.19/edge.
sudo snap install microk8s --classic --channel=1.19/edge
microk8s (1.19/edge) v1.19.4 from Canonical✓ installed
# Warning: iptables-legacy tables present, use iptables-legacy to see them
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
cat /boot/firmware/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash
cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 3 1 1
cpu 2 1 1
cpuacct 2 1 1
blkio 4 1 1
memory 0 117 0
devices 8 108 1
freezer 5 2 1
net_cls 9 1 1
perf_event 6 1 1
net_prio 9 1 1
pids 10 115 1
rdma 7 1 1
Same issue here with Ubuntu 20.04.2 on Raspberry Pi 4. Is any workaround to enable cgroup memory?
Same setup here (20.04.2 on Raspberry Pi 4), with latest 1.19
$ snap info microk8s | tail -1
installed: v1.19.9 (2098) 184MB classic
$ sudo lshw | grep Rasp
product: Raspberry Pi 4 Model B Rev 1.2
Adding
cgroup_enable=memory cgroup_memory=1
to /boot/firmware/cmdline.txt and rebooting worked for me.
$ microk8s inspect
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-control-plane-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting juju
Inspect Juju
Inspecting kubeflow
Inspect Kubeflow
Building the report tarball
Report tarball is at /var/snap/microk8s/2098/inspection-report-20210328_001211.tar.gz
I have enabled cgroups memory and microk8s does not want to see it.
# cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 0 360 1
cpu 0 360 1
cpuacct 0 360 1
blkio 0 360 1
memory 0 360 1
devices 0 360 1
freezer 0 360 1
net_cls 0 360 1
perf_event 0 360 1
net_prio 0 360 1
hugetlb 0 360 1
pids 0 360 1
rdma 0 360 1
# microk8s inspect
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-control-plane-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting juju
Inspect Juju
Inspecting kubeflow
Inspect Kubeflow
# Warning: iptables-legacy tables present, use iptables-legacy to see them
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
Building the report tarball
Report tarball is at /var/snap/microk8s/2213/inspection-report-20210601_104319.tar.gz
My OS is Debian Sid
I have enabled cgroups memory and microk8s does not want to see it.
# cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpuset 0 360 1 cpu 0 360 1 cpuacct 0 360 1 blkio 0 360 1 memory 0 360 1 devices 0 360 1 freezer 0 360 1 net_cls 0 360 1 perf_event 0 360 1 net_prio 0 360 1 hugetlb 0 360 1 pids 0 360 1 rdma 0 360 1My OS is Debian Sid
You probably didn't.
Did anyone tried adding cgroup_enable=memory cgroup_memory=1 into /etc/default/grub to GRUB_CMDLINE_LINUX_DEFAULT then reinstalling grub with update-grub?
Also, try cat /proc/cmdline to see if the option got there.
Yes, I tried. Output of microk8s inspect attached.
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.10.0-7-amd64 root=/dev/mapper/Rennschnecke-root ro cgroup_enable=memory cgroup_memory=1 quiet
$ microk8s inspect
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-kubelite is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting juju
Inspect Juju
Inspecting kubeflow
Inspect Kubeflow
# Warning: iptables-legacy tables present, use iptables-legacy to see them
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
Building the report tarball
Report tarball is at /var/snap/microk8s/2210/inspection-report-20210622_082450.tar.gz
inspection-report-20210622_082450.tar.gz
@mkesper i can see based on the inspect output, that the cluster seems to be in working condition.
Perhaps the microk8s output is the bug, which does not affect the cluster operation.
It seems so:
$ mount|grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
/src/upstream/microk8s (master) $ rg cgroup
...
scripts/inspect.sh
186: if ! mount | grep -q 'cgroup/memory'; then
187: printf -- '\033[0;33mWARNING: \033[0m The memory cgroup is not enabled. \n'
188: printf -- 'The cluster may not be functioning properly. Please ensure cgroups are enabled \n'
About cgroup error, my solution is add cgroup_enable=memory cgroup_memory=1 into the cmdline.txt of boot partition in SDCard, it works, and my operating system is rpios 64.
But the problem of iptables-legacy tables present, use iptables-legacy to see them remains unresolved.
I've been trying to get microk8s 1.21/stable working on Ubuntu 21.10-arm64 on Pi-4 and it seems that perhaps microk8s doesn't like the new default in Ubuntu 21.10 ( from the release notes )
systemd is being switched to the “unified” cgroup hierarchy (cgroup v2) by default. If for some reason you need to keep the legacy cgroup v1 hierarchy, you can select it via a kernel parameter at boot time: systemd.unified_cgroup_hierarchy=0 (bug 1850667 29)
So - after adding systemd.unified_cgroup_hierarchy=0 the issue with the memory cgroup error, goes away.
(don't forget to rebuild the initramfs! sudo update-initramfs -u )
To see which ( V1 or V2 ) you have configured -
stat -c %T -f /sys/fs/cgroup
V1 — tmpfs
V2 — cgroup2fs
If you get errors about iptables or iptables-legacy ( it seems that installing the microk8s snap installs these rules anyway ) these two commands will clear the tables.
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X
sudo iptables-legacy -F && sudo iptables-legacy -t nat -F && sudo iptables-legacy -t mangle -F && sudo iptables-legacy -X
systemd, strikes again.
I've been trying to get microk8s 1.21/stable working on Ubuntu 21.10-arm64 on Pi-4 and it seems that perhaps microk8s doesn't like the new default in Ubuntu 21.10 ( from the release notes )
systemd is being switched to the “unified” cgroup hierarchy (cgroup v2) by default. If for some reason you need to keep the legacy cgroup v1 hierarchy, you can select it via a kernel parameter at boot time: systemd.unified_cgroup_hierarchy=0 (bug 1850667 29)
So - after adding
systemd.unified_cgroup_hierarchy=0the issue with the memory cgroup error, goes away. (don't forget to rebuild the initramfs!sudo update-initramfs -u) ...systemd, strikes again.
This is not a long time solution but a workaround for the fact that microk8s (at least 1.21 which isn't the most up to date anymore) does not support cgroup v2 properly and uses iptables-legacy for its whole network configuration. NB: The cluster worked for me apparently with cgroup v2 but there was this irritating message.
Problem present on my system as well. Looks like the inspect test is faulty.
cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS"
cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 0 206 1
cpu 0 206 1
cpuacct 0 206 1
blkio 0 206 1
memory 0 206 1
devices 0 206 1
freezer 0 206 1
net_cls 0 206 1
perf_event 0 206 1
net_prio 0 206 1
hugetlb 0 206 1
pids 0 206 1
rdma 0 206 1
misc 0 206 1
microk8s inspect
<snip>
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.0-41-generic root=UUID=80...16 ro zswap.enabled=1 zswap.compressor=zstd cgroup_enable=memory cgroup_memory=1 cgroup_enable=cpuset
snap list | grep microk8s
microk8s v1.21.12 3202 1.21/stable canonical** classic
uname -a
Linux muon 5.15.0-41-generic #44-Ubuntu SMP Wed Jun 22 14:20:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Hello, I'm running Ubuntu 22.04.1 on arm64 (m1 in parallels) and in my case /boot/cmdline.txt or /boot/firmware/cmdline.txt simply don't exist. Any ideas where I'm supposed to be adding the settings related to cgroups? I'm encoutering the exact same issue when trying to use microk8s 1.19. @hkmaly I have tried enabling the cgroup memory by adding it to the kernel parameters in grub, but it doesn't make any difference.
I did switch to cgroup v1, but take a really long time to reinstall microk8s or to get information about the node (initially), so I'm rather sceptical of this solution.
in my case /boot/cmdline.txt or /boot/firmware/cmdline.txt simply don't exist. Any ideas where I'm supposed to be adding the settings related to cgroups?
I have tried enabling the cgroup memory by adding it to the kernel parameters in grub, but it doesn't make any difference.
There. I mean, those are kernel parameters, the cmdline.txt is just way how to get it to grub. If you are not sure what kernel parameters actually got into running kernel, look into /proc/cmdline.
(Note: I gave up on kubernetes for now and hope that when I will have time for them again it will be in better shape.)
@hkmaly you can always use a higher version though. It works without any issues. This is a special case.
@lethargosapatheia ... higher version of what? You mean like microk9s? :smile:
Here is a fix for this in ubuntu 22.04. Open the file /etc/default/grub in an editor. Find where the string GRUB_CMDLINE_LINUX is set. Add cgroup_enable=memory cgroup_memory=1 to that string. Save the file and exit the editor. Then run sudo update-grub. Here is how my grub file looks like:
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX="cgroup_enable=memory cgroup_memory=1 swapaccount=1 systemd.unified_cgroup_hierarchy=0"
Then you may restart your computer. Then run microk8s.inspect, everything should be ok. Below is my microk8s version
MicroK8s v1.25.2 revision 4055
GRUB_CMDLINE_LINUX="cgroup_enable=memory cgroup_memory=1 systemd.unified_cgroup_hierarchy=0"
I needed all three of these.
same issue installed on Ubuntu core 22.04 LTS - Raspberry pi
cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
mount |grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
microk8s version
MicroK8s v1.25.2 revision 4054
sudo microk8s inspect
WARNING: The memory cgroup is not enabled.
The cluster may not be functioning properly. Please ensure cgroups are enabled
See for example: https://microk8s.io/docs/install-alternatives#heading--arm
Building the report tarball
Report tarball is at /var/snap/microk8s/4054/inspection-report-20221023_225317.tar.gz
