canonical
Node status incorrect after host fail [1.34]
Summary
I'm running 3 nodes on 1.34. When I take a host down I get 1 of 3 outcomes:
- All nodes remain reported as Ready even though a host is not accessible
- The node I put offline will go NotReady as will another healthy one, ie 2 nodes reported as NotReady
- The node I put offline will be marked NotReady
This is reproducible on different systems on both Red Hat and Ubuntu Seeing other undesirable behaviour like pods not being (re)scheduled
dmhost1 taken offline:
labuser@dmhost2:~$ kubectl get no
NAME STATUS ROLES AGE VERSION
dmhost1 NotReady
When I brought host1 back online, it returned to Ready, however host3 did not (should not have been NotReady)
dmhost3 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message
NetworkUnavailable False Thu, 23 Oct 2025 10:45:20 +0000 Thu, 23 Oct 2025 10:45:20 +0000 CalicoIsUp Calico is running on this node MemoryPressure Unknown Thu, 23 Oct 2025 10:51:25 +0000 Thu, 23 Oct 2025 10:50:53 +0000 NodeStatusUnknown Kubelet stopped posting node status. DiskPressure Unknown Thu, 23 Oct 2025 10:51:25 +0000 Thu, 23 Oct 2025 10:50:53 +0000 NodeStatusUnknown Kubelet stopped posting node status. PIDPressure Unknown Thu, 23 Oct 2025 10:51:25 +0000 Thu, 23 Oct 2025 10:50:53 +0000 NodeStatusUnknown Kubelet stopped posting node status. Ready Unknown Thu, 23 Oct 2025 10:51:25 +0000 Thu, 23 Oct 2025 10:50:53 +0000 NodeStatusUnknown Kubelet stopped posting node status.
Seen other strange behaviour like not reacting to the NotReady
What Should Happen Instead?
Failed node, and only failed node, should be marked NotReady
Reproduction Steps
- 3 node system
- Take host down
Introspection Report
dmhost1 offline
inspection-report-20251023_113929.tar.gz
Can you suggest a fix?
Are you interested in contributing with a fix?
Other scenario |(different cluster with just mK8s installed)
Host 1 offline. All nodes reporting Ready
labuser@dailybuild4-host2:~$ kubectl get no
NAME STATUS ROLES AGE VERSION
dailybuild4-host1 Ready
Describe node shows everything OK and ready status being posted, even though the host is offline and not contactable. Looks stale.
CreationTimestamp: Thu, 23 Oct 2025 12:59:03 +0100
Taints:
NetworkUnavailable False Thu, 23 Oct 2025 13:30:24 +0100 Thu, 23 Oct 2025 13:30:24 +0100 CalicoIsUp Calico is running on this node MemoryPressure False Thu, 23 Oct 2025 18:08:30 +0100 Thu, 23 Oct 2025 12:59:03 +0100 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 23 Oct 2025 18:08:30 +0100 Thu, 23 Oct 2025 12:59:03 +0100 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 23 Oct 2025 18:08:30 +0100 Thu, 23 Oct 2025 12:59:03 +0100 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Thu, 23 Oct 2025 18:08:30 +0100 Thu, 23 Oct 2025 13:31:52 +0100 KubeletReady kubelet is posting ready status Addresses: InternalIP: 10.173.130.211 Hostname: dailybuild4-host1
Lease had long since expired (1.5 hours ago)
labuser@dailybuild4-host2:~$ kubectl describe lease dailybuild4-host1 -n kube-node-lease
Name: dailybuild4-host1
Namespace: kube-node-lease
Labels:
inspect attached
3 Ready (1 host down)
inspection-report-20251023_181809.tar.gz
Hi @kcarson77 , I've raised a PR. please review it and suggest me any changes. Thanks :)
thanks - I've done some testing. If I manually add those parameters to the files I get API server instability and flapping leading to cyclic connection failures. I noticed the parameters are not present in the 1.32 files but the defaults seem stable and I don't have these issues, just the known issue around the DB lock (which is why I need 1.34) Feels like some dqlite issue or control plane coordination issue in 1.34
Hi - are there any updates here? If I add the suggested config I get API server instability, in that it continually restarts.
We've had to rollback to 1.32, but that release also has issues. Is the recommendation to rollback to 1.29?
thanks
Also seeing this issue and the fix as suggested (adding the params manually to config) causes a restart as above. Any further guidance would be appreciated as we are stuck here.
Hey @kcarson77 and folks,
I've taken a look into this, however on 1.34 with a 3 node cluster I can not reproduce the issue.
From the inspection report you've provided I observe unusual error messages, is there a possibility for loss of network on hosts that are supposed to be healthy?
Oct 14 14:34:48 dmhost2 microk8s.daemon-k8s-dqlite[3379070]: time="2025-10-14T14:34:48Z" level=warning msg="[go-dqlite] attempt 3181: server 10.173.128.165:19001: dial: dial tcp 10.173.128.165:19001: connect: network is unreachable"
Oct 14 14:34:48 dmhost2 microk8s.daemon-k8s-dqlite[3379070]: time="2025-10-14T14:34:48Z" level=warning msg="[go-dqlite] attempt 3181: server 10.173.128.164:19001: dial: dial tcp 10.173.128.164:19001: connect: network is unreachable"
Oct 14 14:34:48 dmhost2 microk8s.daemon-k8s-dqlite[3379070]: time="2025-10-14T14:34:48Z" level=warning msg="[go-dqlite] attempt 3181: server 10.173.128.166:19001: dial: dial tcp 10.173.128.166:19001: connect: network is unreachable"
I did take 1 node down to trigger this. Sometimes all 3 nodes are "Ready" sometimes 2 NotReady when I take a node down. Seeing this across different systems. SQA reported very occasionally microK8s 1.34 will terminate but not reschedule pods. I'll reinstall microK8s 1.34 and reproduce again
I re-installed 1.34. I have 3 hosts dmhost1, dmhost2 and dmhost3. I downed the link on host3, so that host1 cannot ping it. MicroK8s still has all three nodes "Ready" even though it cannot be contacted. dmhost1 and dmhost2 can communicate, dmhost3 is offline by link down. Expect dmhost3 would be NotReady however still "Ready" after hours.
labuser@dmhost1:~$ ping dmhost3
PING dmhost3 (10.173.128.166) 56(84) bytes of data.
From dmhost1 (10.173.128.164) icmp_seq=1 Destination Host Unreachable
From dmhost1 (10.173.128.164) icmp_seq=2 Destination Host Unreachable
From dmhost1 (10.173.128.164) icmp_seq=3 Destination Host Unreachable
From dmhost1 (10.173.128.164) icmp_seq=4 Destination Host Unreachable
From dmhost1 (10.173.128.164) icmp_seq=5 Destination Host Unreachable
^C
--- dmhost3 ping statistics ---
6 packets transmitted, 0 received, +5 errors, 100% packet loss, time 5064ms
pipe 3
labuser@dmhost1:~$ kubectl get no
NAME STATUS ROLES AGE VERSION
dmhost1 Ready
I can ping dmhost2 labuser@dmhost1:~$ ping dmhost2 PING dmhost2 (10.173.128.165) 56(84) bytes of data. 64 bytes from dmhost2 (10.173.128.165): icmp_seq=1 ttl=64 time=0.107 ms 64 bytes from dmhost2 (10.173.128.165): icmp_seq=2 ttl=64 time=0.072 ms 64 bytes from dmhost2 (10.173.128.165): icmp_seq=3 ttl=64 time=0.110 ms
Ubuntu 22, Everything is vanilla although it is hardened Security hardened system using ubuntu2004_cis-1.0.3
From the describe node it looks like the offline node timers are stale. Still reporting "Ready" but from the time I took the interface down.
Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message
NetworkUnavailable False Thu, 11 Dec 2025 12:54:09 +0000 Thu, 11 Dec 2025 12:54:09 +0000 CalicoIsUp Calico is running on this node MemoryPressure False Thu, 11 Dec 2025 22:58:14 +0000 Thu, 11 Dec 2025 13:25:28 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 11 Dec 2025 22:58:14 +0000 Thu, 11 Dec 2025 13:25:28 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 11 Dec 2025 22:58:14 +0000 Thu, 11 Dec 2025 13:25:28 +0000 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Thu, 11 Dec 2025 22:58:14 +0000 Thu, 11 Dec 2025 13:25:28 +0000 KubeletReady kubelet is posting ready status
I have uploaded the inspection report from dmhost1.
inspection-report-20251211_222413.tar.gz