Unable to run microk8s kubectl after host reboot (Hyper-V)

[doggy8088]

Summary

I have a fully functional MicroK8s running in my Hyper-V. After my host rebooted, I can't use microk8s kubectl anymore. I always get the following error:

microk8s kubectl get node -o wide

Unable to connect to the server: dial tcp 172.31.119.125:16443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

As I know, the master node IP been changed. If I update the KUBECONFIG locally, I can connect to cluster without problem.

microk8s config > ~/.kube/config

But if I want to use microk8s kubectl get node -o wide to get node status, I always can't get it working. I'm still unable to connect to the server.

I tried to clear all possible cache by removing all .kube/cache folders. But still not working.

sudo rm -rf /.kube/cache /root/.kube/cache /home/ubuntu/.kube/cache /var/snap/microk8s/3582/.kube/cache

I stopped and started MicroK8s again. I'm still unable to connect to the server.

microk8s stop
microk8s start

After MicroK8s restarted, I also tried to find out all files that contains 172.31.119.125 ip address.

grep '172.31.119.125' -r /

Nothing useful found. Only /var contains some logs with 172.31.119.125. That's so weird. Is there anything I can try? How to connect to MicroK8s using microk8s kubectl?

What Should Happen Instead?

I expected there a RIGHT way to use microk8s kubectl. Even I reboot the host.

Reproduction Steps

1. microk8s install
2. Reboot host
3. microk8s kubectl get no -o wide

Introspection Report

inspection-report-20220814_104341.tar.gz

Can you suggest a fix?

After 1 hours deep dive, I finally found that there is a $env:LOCALAPPDATA\MicroK8s\config file exist which the doc never said.

PowerShell

microk8s config > $env:LOCALAPPDATA\MicroK8s\config

Command Prompt

microk8s config > %LOCALAPPDATA%\MicroK8s\config

Are you interested in contributing with a fix?

I strongly recommend this information can add to the doc. Thanks. Can I contribute docs?

[madalinignisca]

If I remember right, with Hyper-V you have to set static IPs, as each reboot will make your virtual machines get another ip.

[doggy8088]

@madalinignisca Yes, you are right.

[karolswdev]

would you be able to post how you set up the static IP, please?

[doggy8088]

@karolswdev I'm not setting up static IP address. It because microk8s install use Default Switch as the default network on Hyper-V on Windows which means the IP address will change after reboot.

[madalinignisca]

It can't be fully automated, but config things are short to be done.

@karolswdev here's my usage:

In Hyper-V I am creating a new Virtual switch, simply named multipass.

When I want to create new multipass instances, I am adding as a param to add an additional network interface, example:

multipass launch -n microk8s-3 -c 2 -m 4G -d 40G --network name=multipass,mode=manual

This will launch a new instance that will have a secondary network interface (I use this setup because this matches 99% what I have out of the box in Hetzner, with private network, except in this case, i have to manually add the netplan yaml file)

In the instance, run: ip l and note the generated MAC of ETH1 (copy paste in some notepad)

Create a netplan config: sudo nano /etc/netplan/99-multipass.yaml

Use this template to add your permanent ip. The only important thing is that your machines use identical netmask, so the ip class is not important if it is different from groups of instances to others, as the added vswith is like a dump simple hardware network switch in this case.

ubuntu@microk8s-1:~$ cat /etc/netplan/99-multipass.yaml
network:
    ethernets:
        eth1:
            dhcp4: false
            match:
                macaddress: 52:54:00:d4:44:41
            set-name: eth1
            addresses: [10.0.1.1/16]
    version: 2

I used 10.0.0.0/16 for the instances, as for some experiments with no tunnel networks for pods Microk8s does 10.1.0.0/16 and I am matching this setup in Hetzner with their private network and automatic routes creation using the hetzner ccm for kubernetes (this is another topic, but for me this way of using Hyper-V allowed me to replicate the Hetzner Cloud locally and experiment without expenses). Any other projects outside of MicroK8S, I use other ip address classes.

Run: sudo netplan apply

This will freeze the CLI. If u leave it like this a couple of minutes it will return to your CMD prompt, or if u are impatient, close current windows terminal tab, open another one and shell in the instance again. Test now with : ip a and you should see ETH1 configured.

Enjoy!