Creating tokens for v1.24.2 doesn't appear to work

[dwarf-king-hreidmar]

Before creating an issue, make sure you've checked the following:

• [X] You are running the latest released version of k0s
• [X] Make sure you've searched for existing issues, both open and closed
• [X] Make sure you've searched for PRs too, a fix might've been merged already
• [X] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Version

v1.24.2 +k0s.0

Platform

Distributor ID:	Ubuntu
Description:	Ubuntu 22.04 LTS
Release:	22.04
Codename:	jammy

What happened?

A new kublet is failing to get the initial kubet config with a join token I'm creating. Error: failed to get kublet config from API: Unauthorized" I deleted an existing node when the hostname changed by stopping all the services on the node. I also deleted the node using kubectl on the controller. I generated a new token using : /opt/k0s/bin/k0s token create --role worker --expiry 5m --data-dir /opt/k0s/data. A token is generated

Steps to reproduce

1. Stop k0sworker on existing node
2. Delete k0s directories
3. Delete k0sworker unit file
4. Delete node from controller with kubectl delete node
5. Generate a new key with /opt/k0s/bin/k0s token create --role worker --expiry 5m --data-dir /opt/k0s/data
6. Copy key to a file on the worker node
7. try and install the worker with: /opt/k0s/bin/k0s install worker --data-dir /opt/k0s/data --token-file /opt/k0s/data/join-token
8. Get unauthorized errors above

Expected behavior

The node registers with the controller

Actual behavior

The node fails to register and then shuts down.

Screenshots and logs

No response

Additional context

The hostname of this node changed (from a short name to a long name) and that's why I'm reinstalling. (there were some errors mounting some pvc resources about the resource not being associated with the node after the name changed so figured I needed a fresh start)

I used to use --config with the token create command because I keep my config in a non standard place /opt/k0s/config. I understand this option was removed in the latest version. Not sure if this is contributing to the issue or if the token is bogus or what

[dwarf-king-hreidmar]

I was able to register the node after I installed linux-modules-extra-raspi. It took me a long time to figure this out because I was getting the registration errors above. The logs contain this line

level=warning msg="failed to load nf_conntrack kernel module: /usr/sbin/modprobe nf_conntrack"

is it possible to make this an error if it would prevent registration of a new node?

The fact that I got the unauthorized errors for registration is a red herring and confusing I think.

[twz123]

Hi, I have a few more questions about your setup so I can better understand the problem.

1. You mentioned that you removed and readded a node. That means that joining a cluster worked at least once before. Did you perform a k0s update, or an OS update?
2. You mentioned that you installed linux-modules-extra-raspi, so I assume you're on a Raspberry Pi? The k0s docs mention that nf_conntrack should be loaded, although they don't mention that package. Maybe that changed between Ubuntu 20.04 and 22.04.
3. The connection between the unauthorized error and the solution to provide the nf_conntrack kernel module are not entirely clear to me. Would you be able to provide some more details about the cluster setup? How many nodes, which k0s configuration and version? Ideally some k0s logs.

Runtime dependencies of Kubernetes components are not too easy to diagnose. Often they are not explicitly stated in upstream documentation and rely on tribal knowledge, they are often not a hard requirement in all cases, but only for certain setups. Some are only required for nodes that are running workloads. Moreover, Linux distros tend to be quite diverse, and detecting the requirements at runtime is not guaranteed to provide correct results on all setups.

So we are trying to be conservative about making such things hard failures. We do have the k0s sysinfo command which tries to inspect the current system configuration and might give hints about bad/missing configs (including nf_conntrack). I'd also invite you to have a look at our runtime dependency documentation page. There's probably stuff to be clarified/added. Feel free to open issues for those things.

When I have more infos about your setup, I'll try to reproduce the problem. I think there are several things in here:

1. Was the unauthorized error you saw a direct consequence of the missing nf_conntrack kernel module? In any case, is there something that we can do to provide an error message that provides a better context?
2. Probably the Raspberry Pi docs need an update for the current Ubuntu LTS version.
3. Figure out what exactly fails if nf_conntrack is unavailable, and consider it to make it a hard requirement in the pre-flight checks.

[dwarf-king-hreidmar]

I'll explain how I got here:

1. I had a working cluster with a master and two worker nodes.
2. I had a power outage and all my nodes came back up. However one of my worker nodes showed up as "Not Ready".
3. I figure it could have been a kernel update or anything but wasn't sure and started to dig in. I saw something in the logs with the short name and when I checked hostnamectl it returned the short name HOWEVER the nodes were registered with their long names which perplexed me. I also noticed a ton of csi errors because I use longhorn and there were a bunch of orphaned pods on that host (bug in longhorn causes this).
4. I cleaned up all the orphaned folders and those errors cleared up
5. I changed the hostname of my node and it still wouldn't connect again.
6. I deleted the keys and tried to register again but I kept getting permission errors
7. I deleted the entire k0s data folder and got a newer version of k0s since I was there anyway.
8. While I was at it I upgraded from 21.xx to 22.04 (to make sure I wouldn't have compat issues)
9. I upgraded the k0s contorller
10. I tried registering the "new" k0s node (the one I deleted the k0s worker from)
11. Still no joy and saw the errors above. I didn't know what to do because I was too focused on the token denied errors. I struggled a bit thinking there might be a new registration method or that setting a custom data dir wasn't allowed anymore (all wrong)
12. I went up higher in the logs and saw the nf_conntrack log. I noticed that the linux-modules-extra package wasn't there. (I had installed it a long while ago when I initially stood up this cluster it was even in my ansible)
13. As soon as I installed linux-modules-extra registration started working

Sorry this was so long winded. I tried to capture as much detail as possible.

I have a theory that before I ever upgraded the OS it may have patched the kernel and I had the kernel version specific linux-modules-extra installed. Maybe it got removed and when they rebooted everything just stopped working. But because I got off on the hostname tangent and re-installed k0s on the node I just hit a new problem. i can't explain the host name changes. I regularly update / reboot. Whats wild is only one worker node refused to connect.

[twz123]

Hey @dwarf-king-hreidmar,

if I understood all of the above correctly, the core issue was that a k0s worker node was not able to start, because of the missing nf_conntrack kernel module. I'd say that this was a general failure and not connected to the fact that you tried to join that node. Probably a stand alone controller with --single or --enable-worker wouldn't have worked either. I've opened #2069 to investigate what we can do to make this problem easier to detect.

I also had a closer look at Ubuntu 22.04.1, both on amd64 and Raspberry Pi 4. The kernels shipped with those versions of Ubuntu work just fine with k0s. Both include the nf_conntrack module in the default install. I cannot say why it was missing in your installation.

Concerning the other issue about the hostnames: I'm a bit curious. Not sure what you mean by long vs. short hostnames. Mind providing an example? K0s doesn't have a flag to override the hostname, so if that changes (according to what uname reports), the k8s node identity changes with it.

[dwarf-king-hreidmar]

@twz123 Did you check the desktop or server version? I have the server version installed. This node was also upgraded from an older version of Ubuntu. Maybe when I upgraded from one major version to another it didn't carry over due to a bug or something.

Concerning hostnames. I don't think hostnames actually mattered. Without any other clear reason for the failed to get kublet config from API: Unauthorized" error I started investigating everything I could. I noticed that kubectl get nodes still had my node listed :k3snode-1.domain.local NotReady <none> 27d v1.24.2+k0s however hostname -f reported: k3snode-1 . So i thought maybe I was unauthorized because the host name mismatch. All my nodes were registered wit their fqdn. Sounds like it wouldn't have mattered.

I defer to you folks but the only suggestion I had was to make the nf_conntrackwarning an error as I'm pretty sure nothing is gonna work correctly if you get that warning

[twz123]

@twz123 Did you check the desktop or server version? I have the server version installed. This node was also upgraded from an older version of Ubuntu. Maybe when I upgraded from one major version to another it didn't carry over due to a bug or something.

I tested Ubuntu Server. The preinstalled server image for the Pi and the minimal server install for amd64. I'd also guess that the missing modules were connected to the update process not installing the same packages as a fresh install does.

Concerning hostnames. I don't think hostnames actually mattered. Without any other clear reason for the failed to get kublet config from API: Unauthorized" error I started investigating everything I could. I noticed that kubectl get nodes still had my node listed :k3snode-1.domain.local NotReady <none> 27d v1.24.2+k0s however hostname -f reported: k3snode-1 . So i thought maybe I was unauthorized because the host name mismatch. All my nodes were registered wit their fqdn. Sounds like it wouldn't have mattered.

Yeah, unrelated to the "Unauthorized" problem. Nevertheless I wanted to outline that whatever the kernel reports as a hostname will be used as Kubernetes node name.

I defer to you folks but the only suggestion I had was to make the nf_conntrackwarning an error as I'm pretty sure nothing is gonna work correctly if you get that warning

This probably only applies to the container network components (i.e. kube-proxy and friends). A bare k0s controller without worker components might work just fine without nf_conntrack. Anyhow, the UX for this problem could be much better, making troubleshooting easier.

[thesn10]

I'd also guess that the missing modules were connected to the update process not installing the same packages as a fresh install does.

I have the same problem and i did not upgrade from an older version of ubuntu. My ubuntu version is Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-124-generic x86_64)

failed to load nf_conntrack kernel module: /usr/sbin/modprobe nf_conntrack

I am new to k0s and installed it the first time and got this error. I was just following the Get-Started guide with following commands:

> curl -sSLf https://get.k0s.sh | sudo sh

> k0s install controller --single --enable-worker

> k0s status
k0s status
Version: v1.24.3+k0s.0
Process ID: 1359060
Role: controller
Workloads: true
SingleNode: true

> k0s kubectl get nodes
No resources found

You can see that it doesnt even show any nodes and in status command doesnt give the worker role: Role: controller instead of Role: controller+worker

I also get the waring:

failed to load nf_conntrack kernel module: /usr/sbin/modprobe nf_conntrack

Here are the full logs:

> journalctl -u k0scontroller --no-pager | grep 'error\|warning'

Aug 22 15:40:02 v2202007125438122816 k0s[920]: time="2022-08-22 15:40:02" level=warning msg="exit status 1" component=kube-controller-manager
Aug 22 15:40:02 v2202007125438122816 k0s[920]: time="2022-08-22 15:40:02" level=warning msg="exit status 1" component=kube-scheduler
Aug 22 15:40:03 v2202007125438122816 k0s[920]: time="2022-08-22 15:40:03" level=error msg="failed to stop component Status: remove /run/k0s/status.sock: no such file or directory"
Aug 22 15:40:08 v2202007125438122816 k0s[920]: time="2022-08-22 15:40:08" level=error msg="Failed to stop node components" error="failed to stop components"
Aug 22 15:43:19 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:19" level=warning msg="datasource file /var/lib/k0s/db/state.db does not exist"
Aug 22 15:43:25 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:25" level=warning msg="Extensions CRD is not yet ready, waiting before starting ExtensionsController" component=extensions_controller
Aug 22 15:43:25 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:25" level=warning msg="Extensions CRD is not yet ready, waiting before starting ExtensionsController" component=extensions_controller
Aug 22 15:43:30 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:30" level=warning msg="Extensions CRD is not yet ready, waiting before starting ExtensionsController" component=extensions_controller
Aug 22 15:43:37 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:37" level=warning msg="failed to load nf_conntrack kernel module: /usr/sbin/modprobe nf_conntrack"
Aug 22 15:43:38 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:38" level=warning msg="failed to get initial kubelet config with join token: failed to get kubelet config from API: configmaps \"kubelet-config-default-1.24\" is forbidden: User \"system:bootstrap:rjx6bf\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\""
Aug 22 15:43:38 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:38" level=warning msg="exit status 1" component=kube-controller-manager
Aug 22 15:43:38 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:38" level=warning msg="failed to get initial kubelet config with join token: failed to get kubelet config from API: configmaps \"kubelet-config-default-1.24\" is forbidden: User \"system:bootstrap:rjx6bf\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\""
Aug 22 15:43:39 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:39" level=warning msg="failed to get initial kubelet config with join token: failed to get kubelet config from API: configmaps \"kubelet-config-default-1.24\" is forbidden: User \"system:bootstrap:rjx6bf\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\""
Aug 22 15:43:41 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:41" level=warning msg="failed to get initial kubelet config with join token: failed to get kubelet config from API: configmaps \"kubelet-config-default-1.24\" is forbidden: User \"system:bootstrap:rjx6bf\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\""
Aug 22 15:43:42 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:42" level=error msg="unable to register 'update' controllers" component=autopilot error="client rate limiter Wait returned an error: context canceled" leadermode=false
Aug 22 15:43:42 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:42" level=error msg="failed to start subhandlers: client rate limiter Wait returned an error: context canceled" component=autopilot leasemode=acquired
Aug 22 15:43:43 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:43" level=warning msg="exit status 1" component=kube-controller-manager
Aug 22 15:43:45 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:45" level=warning msg="failed to get initial kubelet config with join token: failed to get kubelet config from API: configmaps \"kubelet-config-default-1.24\" is forbidden: User \"system:bootstrap:rjx6bf\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"kube-system\""
Aug 22 15:43:49 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:49" level=warning msg="exit status 1" component=kube-controller-manager
Aug 22 15:43:53 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:53" level=warning msg="exit status 1" component=kubelet
Aug 22 15:43:54 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:54" level=warning msg="exit status 1" component=kube-controller-manager
Aug 22 15:43:58 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:58" level=warning msg="Failed to load autopilot client config, retrying in 5s" component=autopilot error="invalid configuration: [unable to read client-cert /var/lib/k0s/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/k0s/kubelet/pki/kubelet-client-current.pem: no such file or directory, unable to read client-key /var/lib/k0s/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/k0s/kubelet/pki/kubelet-client-current.pem: no such file or directory]"
Aug 22 15:43:58 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:43:58" level=warning msg="exit status 1" component=kubelet
Aug 22 15:44:00 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:44:00" level=warning msg="exit status 1" component=kube-controller-manager
Aug 22 15:44:03 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:44:03" level=warning msg="Failed to load autopilot client config, retrying in 5s" component=autopilot error="invalid configuration: [unable to read client-cert /var/lib/k0s/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/k0s/kubelet/pki/kubelet-client-current.pem: no such file or directory, unable to read client-key /var/lib/k0s/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/k0s/kubelet/pki/kubelet-client-current.pem: no such file or directory]"
Aug 22 15:48:50 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:48:50" level=warning msg="exit status 1" component=kubelet
Aug 22 15:48:52 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:48:52" level=warning msg="exit status 1" component=kube-controller-manager
Aug 22 15:48:53 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:48:53" level=warning msg="Failed to load autopilot client config, retrying in 5s" component=autopilot error="invalid configuration: [unable to read client-cert /var/lib/k0s/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/k0s/kubelet/pki/kubelet-client-current.pem: no such file or directory, unable to read client-key /var/lib/k0s/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/k0s/kubelet/pki/kubelet-client-current.pem: no such file or directory]"
Aug 22 15:48:53 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:48:53" level=error msg="Failed to start controller worker" error="failed to start worker components: unable to create autopilot client: timed out waiting for the condition"
Aug 22 15:48:58 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:48:58" level=warning msg="exit status 1" component=kube-controller-manager
Aug 22 15:49:04 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:49:04" level=warning msg="exit status 1" component=kube-controller-manager
Aug 22 15:49:10 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:49:10" level=warning msg="exit status 1" component=kube-controller-manager
Aug 22 15:49:16 v2202007125438122816 k0s[1359060]: time="2022-08-22 15:49:16" level=warning msg="exit status 1" component=kube-controller-manager

[twz123]

@SnGmng did you check if you have the nf_conntrack module available for your kernel? There should be a file called /lib/modules/5.4.0-124-generic/kernel/net/netfilter/nf_conntrack.ko. Does a manual modprobe (sudo modprobe nf_conntrack) work?

I checked the focal packages and that file should be part of the linux-modules package, if I'm not mistaken.

Edit: For your kernel, the package seems to be in the focal-updates repo: https://packages.ubuntu.com/focal-updates/amd64/linux-modules-5.4.0-124-generic/filelist

[thesn10]

@twz123

Yes it is installed and you can see the exit code of modprobe is 0 so its loaded sucessfully

> /usr/sbin/modprobe nf_conntrack && echo $?

0

> ls /lib/modules/5.4.0-124-generic/kernel/net/netfilter/

ipset                       nf_log_netdev.ko        nft_counter.ko       nft_reject.ko      xt_conntrack.ko  xt_length.ko      xt_realm.ko
ipvs                        nf_nat_amanda.ko        nft_ct.ko            nft_socket.ko      xt_cpu.ko        xt_limit.ko       xt_recent.ko
nf_conncount.ko             nf_nat_ftp.ko           nft_dup_netdev.ko    nft_synproxy.ko    xt_CT.ko         xt_LOG.ko         xt_REDIRECT.ko
nf_conntrack_amanda.ko      nf_nat_irc.ko           nft_fib_inet.ko      nft_tproxy.ko      xt_dccp.ko       xt_mac.ko         xt_sctp.ko
nf_conntrack_broadcast.ko   nf_nat.ko               nft_fib.ko           nft_tunnel.ko      xt_devgroup.ko   xt_mark.ko        xt_SECMARK.ko
nf_conntrack_ftp.ko         nf_nat_sip.ko           nft_fib_netdev.ko    nft_xfrm.ko        xt_dscp.ko       xt_MASQUERADE.ko  xt_set.ko
nf_conntrack_h323.ko        nf_nat_tftp.ko          nft_flow_offload.ko  x_tables.ko        xt_DSCP.ko       xt_multiport.ko   xt_socket.ko
nf_conntrack_irc.ko         nfnetlink_acct.ko       nft_fwd_netdev.ko    xt_addrtype.ko     xt_ecn.ko        xt_nat.ko         xt_state.ko
nf_conntrack.ko             nfnetlink_cthelper.ko   nft_hash.ko          xt_AUDIT.ko        xt_esp.ko        xt_NETMAP.ko      xt_statistic.ko
nf_conntrack_netbios_ns.ko  nfnetlink_cttimeout.ko  nft_limit.ko         xt_bpf.ko          xt_hashlimit.ko  xt_nfacct.ko      xt_string.ko
nf_conntrack_netlink.ko     nfnetlink.ko            nft_log.ko           xt_cgroup.ko       xt_helper.ko     xt_NFLOG.ko       xt_tcpmss.ko
nf_conntrack_pptp.ko        nfnetlink_log.ko        nft_masq.ko          xt_CHECKSUM.ko     xt_hl.ko         xt_NFQUEUE.ko     xt_TCPMSS.ko
nf_conntrack_sane.ko        nfnetlink_osf.ko        nft_nat.ko           xt_CLASSIFY.ko     xt_HL.ko         xt_osf.ko         xt_TCPOPTSTRIP.ko
nf_conntrack_sip.ko         nfnetlink_queue.ko      nft_numgen.ko        xt_cluster.ko      xt_HMARK.ko      xt_owner.ko       xt_tcpudp.ko
nf_conntrack_snmp.ko        nf_synproxy_core.ko     nft_objref.ko        xt_comment.ko      xt_IDLETIMER.ko  xt_physdev.ko     xt_TEE.ko
nf_conntrack_tftp.ko        nf_tables.ko            nft_osf.ko           xt_connbytes.ko    xt_ipcomp.ko     xt_pkttype.ko     xt_time.ko
nf_dup_netdev.ko            nf_tables_set.ko        nft_queue.ko         xt_connlabel.ko    xt_iprange.ko    xt_policy.ko      xt_TPROXY.ko
nf_flow_table_inet.ko       nft_chain_nat.ko        nft_quota.ko         xt_connlimit.ko    xt_ipvs.ko       xt_quota.ko       xt_TRACE.ko
nf_flow_table.ko            nft_compat.ko           nft_redir.ko         xt_connmark.ko     xt_l2tp.ko       xt_rateest.ko     xt_u32.ko
nf_log_common.ko            nft_connlimit.ko        nft_reject_inet.ko   xt_CONNSECMARK.ko  xt_LED.ko        xt_RATEEST.ko

[thesn10]

Update: after running k0s reset and reboot, there is no nf_conntrack error anymore, but the worker service still wont start:

time="2022-08-22 22:03:31" level=error msg="Failed to start controller worker" error="failed to start kubelet config client: failed to load kubeconfig: invalid configuration: [unable to read client-cert /var/lib/k0s/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/k0s/kubelet/pki/kubelet-client-current.pem: no such file or directory, unable to read client-key /var/lib/k0s/kubelet/pki/kubelet-client-current.pem for default-auth due to open /var/lib/k0s/kubelet/pki/kubelet-client-current.pem: no such file or directory]"
time="2022-08-22 22:03:31" level=error msg="unable to register 'update' controllers" component=autopilot error="client rate limiter Wait returned an error: context canceled" leadermode=false
time="2022-08-22 22:03:31" level=error msg="failed to start subhandlers: client rate limiter Wait returned an error: context canceled" component=autopilot leasemode=acquired

EDIT: The nf_conntrack error has appeared again

[github-actions[bot]]

The issue is marked as stale since no activity has been recorded in 30 days

[github-actions[bot]]

The issue is marked as stale since no activity has been recorded in 30 days