running microk8s inside an ignite vm causes it to stop (!)

[lukemarsden]

FROM weaveworks/ignite-ubuntu:20.04
RUN apt-get update && \
    apt-get install -y docker.io iptables conntrack
RUN apt-get install -y snapd selinux-utils selinux-policy-default
RUN echo "SELINUX=disabled" > /etc/selinux/config

snap install microk8s --classic

A few moments later, the VM just terminates (goes into Stopped state).

53f9da7e6ffe0c76        testfaster-image:3dc71f03b8f6ba5062240ae6d879492c550faa4bcb2c04cd6f1265323aeed2a0-final quay.io/testfaster/ignite-kernel:latest 50.0 GB 4       16.0 GB 9m49s ago       Stopped                                 tfastpool-8d20e06a7e00ce37158ce4ec040fa6cc43ab918eb7315823f3d8a7746c3de3de-vm-bv9ojh0vjfbsorats7c0

journalctl -f & shows the following suspicious looking lines:

Dec 11 15:08:40 53f9da7e6ffe0c76 systemd-udevd[4505]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 11 15:08:40 53f9da7e6ffe0c76 systemd-udevd[4505]: Using default interface naming scheme 'v245'.
Dec 11 15:08:40 53f9da7e6ffe0c76 systemd-udevd[4505]: calico_tmp_B: Could not generate persistent MAC: No data available
Dec 11 15:08:40 53f9da7e6ffe0c76 systemd-udevd[4512]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 11 15:08:40 53f9da7e6ffe0c76 systemd-udevd[4512]: Using default interface naming scheme 'v245'.
Dec 11 15:08:40 53f9da7e6ffe0c76 systemd-udevd[4512]: calico_tmp_A: Could not generate persistent MAC: No data available

I found https://github.com/systemd/systemd/issues/3374#issuecomment-288882355 and tried adding the following lines:

RUN cp /usr/lib/systemd/network/99-default.link /etc/systemd/network/99-default.link
RUN sed -i 's/MACAddressPolicy=persistent/MACAddressPolicy=none/' \
    /etc/systemd/network/99-default.link

This just makes the last few log lines before the VM stops be:

Dec 11 15:28:41 486608395a84719a systemd-udevd[4477]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 11 15:28:41 486608395a84719a systemd-udevd[4579]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 11 15:28:41 486608395a84719a systemd-udevd[4579]: Using default interface naming scheme 'v245'.

But the issue still occurs. What on earth is going on?

[darkowlzz]

Hi, thanks for reporting the issue. I tried reproducing it by creating a new VM, as per the docs at https://microk8s.io/docs, with 4GB memory and 20GB storage. In the VM, I installed snapd and using snap, installed microk8s as per the same docs.

snap install microk8s --classic --channel=1.19

I didn't install docker. microk8s ran its own containerd as the container runtime.

root@bb9162ea7fddf3d3:~# microk8s kubectl get node -o wide
NAME               STATUS   ROLES    AGE   VERSION                     INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
bb9162ea7fddf3d3   Ready    <none>   41m   v1.19.3-34+a56971609ff35a   10.61.0.4     <none>        Ubuntu 20.04.1 LTS   4.19.125         containerd://1.3.7

I deployed a simple pod with busybox to ensure that the cluster works and it worked as expected:

root@bb9162ea7fddf3d3:~# microk8s kubectl logs hello-vqrv8
Hello, Kubernetes!

I waited for ~50 minutes to see if the VM stops, but it continued to run.

$ sudo ignite ps
VM ID			IMAGE				KERNEL					SIZE	CPUS	MEMORY	CREATED	STATUS	IPS		PORTSNAME
bb9162ea7fddf3d3	weaveworks/ignite-ubuntu:latest	weaveworks/ignite-kernel:4.19.125	20.0 GB	1	4.0 GB	53m ago	Up 53m	10.61.0.4		my-vm

Maybe the issue you're seeing is related to insufficient storage? By default, ignite creates a VM with 512MB of memory and 4GB of storage. It could be another one of the strange behaviors when the VM runs out of space.

[stealthybox]

It could be another one of the strange behaviors when the VM runs out of space.

For debugging you could check this by mounting the ext4 loopfile for the VM locally to examine its fs with your host's tools. (df, du -sh, find, .. etc)

If the vm won't run because there is no writeable storage, you will be able to see this from the filesystem and figure out which directories grew beyond your needs.