siderolabs
kubelet_volume_stats_* metrics not available in 1.34
Bug Report
When using Kubernetes version 1.34 kubelet_volume_stats_* metrics such as kubelet_volume_stats_available_bytes are no longer present
Description
More details here: https://github.com/kubernetes/kubernetes/issues/133847
Issue looks resolved in Kubernetes version 1.34.2
Environment
- Talos version: 1.11.5
- Kubernetes version: 1.34.1
Because of this: https://github.com/siderolabs/talos/blob/release-1.11/pkg/machinery/constants/constants.go#L399
and TalosControlPlane
spec:
version: v1.34.2
not works.
upgrade-k8s will upgrade cluster but Cluster API will keep only what is released in Talos 1.11.5 (v1.34.1)
not sure I follow, what's relation b/w talos and cluster api, talos can be installed with any supported k8s version, don't see any issue here
@frezbo The issue is specific to Cluster API deployments with TalosControlPlane.
When managing Talos clusters via CAPI, the TalosControlPlane spec has a version field that's meant to declare the desired Kubernetes version:
spec:
version: v1.34.2
However, the control-plane provider doesn't actually swap Kubernetes binaries on the nodes—it only tracks the version as metadata for rollout decisions. The actual Kubernetes version running on nodes comes from whatever is bundled in the Talos image.
Since Talos 1.11.5 ships with Kubernetes v1.34.1, setting spec.version: v1.34.2 in the TalosControlPlane resource has no effect—the nodes stay on v1.34.1, which has the missing kubelet_volume_stats_* metrics bug.
Running talosctl upgrade-k8s manually can upgrade to v1.34.2, but CAPI will drift the cluster back to v1.34.1 on the next reconciliation since that's what the Talos 1.11.5 image contains.
To properly fix this in a CAPI-managed cluster, we need a Talos release (e.g., 1.11.6) that bundles Kubernetes v1.34.2.
This explains the CAPI-specific behavior and why it's a Talos packaging issue rather than something you can work around with upgrade-k8s.
this is expected behavior, new nodes would only get updates, 1.11.6 would be released in time for security fixes
so either scale down one node at a time and scale up, or add new nodes if there's available
Looks like Talos 1.11.6 released without Kubernetes version bump
you can run any supported version of Kubernetes with Talos v1.11.6
As I explained above, with CAPI I can’t change it — I was surprised too.
Even if I run talosctl upgrade-k8s, CAPI will still reconcile new/replaced nodes using the Talos image’s built-in Kubernetes version. So after a Talos rollout, newly created nodes come up with DefaultKubernetesVersion = "1.34.1" (until the Talos image is updated to include 1.34.2).
This has nothing to do with Talos default version, you need to figure out what you're doing wrong with CAPI.
Talos machine config should have explicit Kubernetes version defined.