kubeblocks icon indicating copy to clipboard operation
kubeblocks copied to clipboard

Published 20 hours ago • verified publisher icon apecloud

[BUG] Deadlock, when disabling an addon if there is any cluster refers to it

Open shanshanying opened this issue 6 months ago • 2 comments

version

kbcli version
Kubernetes: v1.28.3-aliyun.1
KubeBlocks: 0.9.1-beta.13
kbcli: 0.9.0

How to reproduce:

  1. enable an addon on
  2. create a cluster
  3. disable the addon
  4. delete the cluster

We noticed that the cluster cannot be deleted because the the referenced ClusterDefinition is not up to date Check the referred Clusterdefinition and we found the

  • metadata.generation is 3
  • status.observeredGenerations is 2 and the CD won't be deleted cause there is a cluster refering to it.

Comparing the CD before and after disable Addon, we found the generation is increased from 2 to 3. img_v3_02dp_17ee29a6-8df5-48b8-9761-79d03041028g

A quick workaround is

  • patch the CD's status, e.g. k patch cd milvus-2.3.2 -p'{"status":{"observedGeneration":3}}' --type=merge --subresource status

But we should shoot the root cause first to break the deadlock.

shanshanying avatar Aug 15 '24 13:08 shanshanying