talos icon indicating copy to clipboard operation
talos copied to clipboard

Unable to upgrade k8s to 1.30.1

Open Piccirello opened this issue 2 months ago • 1 comments

Bug Report

I'm running a two node Talos cluster on Proxmox - one control plane node and one worker node. When attempting to upgrade Kubernetes from 1.30.0 to 1.30.1, the upgrade always fails. I've attempted this upgrade when booted from Talos 1.7.0, 1.7.1, 1.7.2, and 1.7.4. Each time, the upgrade times out when attempting to upgrade kubelet on the control plane node.

Logs

$ talosctl upgrade-k8s       
automatically detected the lowest Kubernetes version 1.30.0
discovered controlplane nodes ["192.168.10.11" "192.168.10.11"]
discovered worker nodes ["192.168.10.12"]
 > "192.168.10.11": pre-pulling registry.k8s.io/kube-apiserver:v1.30.1
 > "192.168.10.11": pre-pulling registry.k8s.io/kube-apiserver:v1.30.1
 > "192.168.10.11": pre-pulling registry.k8s.io/kube-controller-manager:v1.30.1
 > "192.168.10.11": pre-pulling registry.k8s.io/kube-controller-manager:v1.30.1
 > "192.168.10.11": pre-pulling registry.k8s.io/kube-scheduler:v1.30.1
 > "192.168.10.11": pre-pulling registry.k8s.io/kube-scheduler:v1.30.1
 > "192.168.10.11": pre-pulling ghcr.io/siderolabs/kubelet:v1.30.1
 > "192.168.10.11": pre-pulling ghcr.io/siderolabs/kubelet:v1.30.1
 > "192.168.10.12": pre-pulling ghcr.io/siderolabs/kubelet:v1.30.1
updating "kube-apiserver" to version "1.30.1"
 > "192.168.10.11": starting update
 > "192.168.10.11": machine configuration patched
 > "192.168.10.11": waiting for kube-apiserver pod update
 < "192.168.10.11": successfully updated
 > "192.168.10.11": starting update
 > "192.168.10.11": machine configuration patched
 > "192.168.10.11": waiting for kube-apiserver pod update
 < "192.168.10.11": successfully updated
updating "kube-controller-manager" to version "1.30.1"
 > "192.168.10.11": starting update
 > "192.168.10.11": machine configuration patched
 > "192.168.10.11": waiting for kube-controller-manager pod update
 > "192.168.10.11": kube-controller-manager: pod is not ready, waiting
 > "192.168.10.11": kube-controller-manager: pod is not ready, waiting
 < "192.168.10.11": successfully updated
 > "192.168.10.11": starting update
 > "192.168.10.11": machine configuration patched
 > "192.168.10.11": waiting for kube-controller-manager pod update
 < "192.168.10.11": successfully updated
updating "kube-scheduler" to version "1.30.1"
 > "192.168.10.11": starting update
 > "192.168.10.11": machine configuration patched
 > "192.168.10.11": waiting for kube-scheduler pod update
 < "192.168.10.11": successfully updated
 > "192.168.10.11": starting update
 > "192.168.10.11": machine configuration patched
 > "192.168.10.11": waiting for kube-scheduler pod update
 < "192.168.10.11": successfully updated
updating kube-proxy to version "1.30.1"
 > "192.168.10.11": starting update
 > "192.168.10.11": starting update
updating kubelet to version "1.30.1"
 > "192.168.10.11": starting update
 > "192.168.10.11": machine configuration patched
 > "192.168.10.11": waiting for node update
failed upgrading kubelet: error updating node "192.168.10.11": 3 error(s) occurred:
	node version mismatch: got "v1.30.0", expected "v1.30.1"
	etcdserver: request timed out
	timeout

Environment

  • Talos version:
$ talosctl version -n 192.168.10.11,192.168.10.12
Client:
	Tag:         v1.7.4
	SHA:         cb3a8308
	Built:       
	Go version:  go1.22.3
	OS/Arch:     darwin/arm64
Server:
	NODE:        192.168.10.11
	Tag:         v1.7.4
	SHA:         cb3a8308
	Built:       
	Go version:  go1.22.3
	OS/Arch:     linux/amd64
	Enabled:     RBAC
	NODE:        192.168.10.12
	Tag:         v1.7.4
	SHA:         cb3a8308
	Built:       
	Go version:  go1.22.3
	OS/Arch:     linux/amd64
	Enabled:     RBAC
  • Kubernetes version:
$ kubectl version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.1
  • Platform: metal-amd64

Piccirello avatar Jun 17 '24 20:06 Piccirello