cluster-api-provider-openstack icon indicating copy to clipboard operation
cluster-api-provider-openstack copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes-sigs

kubernetes cluster upgrade breaks with single-node control plane

Open scrungus opened this issue 2 years ago • 12 comments

/kind bug

What steps did you take and what happened: kubernetes cluster upgrade breaks with single-node control plane - either fails to swap the floating IP over to the new control node or unassigns the floating IP completely

Screenshot 2022-12-14 at 17 04 27 This is a kubernetes cluster with 2 worker nodes and 1 control node that was deployed via helm on a management cluster, using the openstack cluster api provider to spin up machines. It was deployed with kubernetes version v1.24.2. A helm upgrade was performed to v1.25.4. The controller successfully spun up a control node and two worker nodes with the new kubernetes version and deleted the old ones, but failed to swap over the floating IP to the new control node.

What did you expect to happen: the original control node should retain the floating IP until the new one is ready, then the floating IP should be moved over to the new control node

Anything else you would like to add: testing an upgrade from kubernetes v1.24.2 to v1.25.4

Environment:

  • Cluster API Provider OpenStack version (Or git rev-parse HEAD if manually built): on a fork of this repository which is missing everything after SHA: 937b278e53c06308997d97973c4f0260b669ed99

  • Cluster-API version:

  • OpenStack version: Ussuri

  • Minikube/KIND version: -

  • Kubernetes version (use kubectl version): v1.24.7 (management cluster)

  • OS (e.g. from /etc/os-release): Ubuntu 20.04

scrungus avatar Dec 14 '22 17:12 scrungus

I am wondering whether this is same to https://github.com/kubernetes-sigs/cluster-api-provider-openstack/issues/1265

as that issue also talk about

The cause of the issue is quite straightforward: as we don't check if floatingIP is already associated to an healthy control-plane machine, each machine will [attach the floating IP to its port when it is reconcilated.](https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/v0.6.3/controllers/openstackmachine_controller.go#L384).

so which CAPO you are using?

jichenjc avatar Dec 15 '22 01:12 jichenjc

We are running v0.7.0-rc.0

scrungus avatar Jan 17 '23 12:01 scrungus

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Apr 17 '23 13:04 k8s-triage-robot

It's probably too late to ask for logs on this one, but I wonder what the sequence of events here is. Presumably we're going to add the new control plane node first before taking the old one down. In that case I'd expect to see this log message:

https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/34ed24f3726d8b13c2b4a634cf96a739cf286e66/controllers/openstackmachine_controller.go#L412-L415

This doesn't return an error so won't cause another reconcile attempt. If it did return an error, it would likely prevent the old control plane node from being removed. My gut feeling is that this can't work.

I suspect we should move the API load balancer code out of the machine controller and into the cluster controller. Or possibly even its own controller 🤔

/remove-lifecycle stale

mdbooth avatar Apr 18 '23 08:04 mdbooth

/kind bug

Ritikaa96 avatar Jun 06 '23 11:06 Ritikaa96

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jan 22 '24 00:01 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Feb 21 '24 00:02 k8s-triage-robot

/remove-lifecycle rotten

mdbooth avatar Feb 21 '24 11:02 mdbooth

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar May 21 '24 11:05 k8s-triage-robot

/remove-lifecycle stale

mdbooth avatar May 21 '24 14:05 mdbooth

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Aug 19 '24 15:08 k8s-triage-robot

/remove-lifecycle stale

EmilienM avatar Aug 19 '24 15:08 EmilienM

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Nov 17 '24 16:11 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Dec 17 '24 17:12 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-triage-robot avatar Jan 16 '25 17:01 k8s-triage-robot

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot avatar Jan 16 '25 17:01 k8s-ci-robot