kubeone icon indicating copy to clipboard operation
kubeone copied to clipboard
verified publisher icon kubermatic

Kubernetes version upgrade renewed leader control plane kubelet server certificate but not others

Open embik opened this issue 2 years ago • 9 comments

What happened?

A user experienced the following situation:

  • they ran kubeone apply to upgrade to Kubernetes 1.24.9 in January this year
  • apparently, kubelet server certificates for the leader control plane node got rotated at that point in time, so kubeone properly approved a pending CSR for a serving certificate
  • it seems that the follower control plane nodes were also upgraded, but did not rotate their kubelet server certificate (it's not entirely clear why, because this was only noticed two months later)
  • the user got an alert saying that kubelet server certificates will expire soon and realised they weren't rotated for the follower nodes

Now I suspect this could have been solved with kubeone apply --force-upgrade, but I did not want to try that theory out on a production environment.

Expected behavior

kubeone should rotate all kubelet server certificates on a version upgrade.

How to reproduce the issue?

Not clear yet. I'm not sure if I can create certificates with a shorter TTL to reproduce this in a setup.

What KubeOne version are you using?

KubeOne 1.5.6

Provide your KubeOneCluster manifest here (if applicable)

# paste manifest here

What cloud provider are you running on?

N/A

What operating system are you running in your cluster?

N/A

Additional information

This can be manually fixed by restarting kubelets and approving most recent CSRs via kubectl approve.

embik avatar Mar 15 '23 12:03 embik

@embik Do you have some idea if this is some cloud provider or baremetal?

xmudrii avatar Mar 15 '23 12:03 xmudrii

@xmudrii Cloud provider is OpenStack

jwerner-mt avatar Mar 15 '23 14:03 jwerner-mt

Issues go stale after 90d of inactivity. After a furter 30 days, they will turn rotten. Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

kubermatic-bot avatar Jul 23 '23 11:07 kubermatic-bot

/remove-lifecycle stale

xmudrii avatar Jul 23 '23 12:07 xmudrii

Issues go stale after 90d of inactivity. After a furter 30 days, they will turn rotten. Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

kubermatic-bot avatar Oct 21 '23 23:10 kubermatic-bot

/remove-lifecycle stale

embik avatar Oct 22 '23 12:10 embik

Issues go stale after 90d of inactivity. After a furter 30 days, they will turn rotten. Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

kubermatic-bot avatar Jun 11 '24 14:06 kubermatic-bot

/remove-lifecycle stale

embik avatar Jun 11 '24 14:06 embik

@kron4eg Can you write down steps how to come up with a setup where we can renew certificates? We can drop the refinement-needed label

xmudrii avatar Jun 27 '24 12:06 xmudrii