`kamardactl unjoin cluster_name` still deleted propagated resources

[icy]

Please provide an in-depth description of the question you have:

I want my cluster to be disjoined but the propagated resources are still there. However, the following command still ends up in deleting all my resources:

$ karmadactl get deployment -n test
nginx-deployment   dr3c1        0/0     0            0           25s   Y
nginx-deployment   staging-sg   6/6     6            6           16m   Y

$ kubectl_for_dr3c1 get namespace test
NAME   STATUS   AGE
test   Active   12s

$ karmadactl unjoin --help
Examples:                                                                            
  # Unjoin cluster from karmada control plane, but not to remove resources created by karmada in the unjoining cluster
  kubectl karmada unjoin CLUSTER_NAME                                                                                                                                     

$ kamardactl unjoin dr3c1 
$ kubectl_for_dr3c1 get namespace test
Error from server (NotFound): namespaces "test" not found

My expectation is that the namespace test still remains after the cluster is disjoined.

What do you think about this question?:

Environment:

• Karmada version:

$ karmadactl version
karmadactl version: version.Info{GitVersion:"v1.7.1", GitCommit:"85fd8c1046c01da340d78325af47d2e944991809", GitTreeState:"clean", BuildDate:"2023-10-24T03:24:01Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes version:

NAME         VERSION    MODE   READY   AGE
dr3c1        v1.25.12   Push   True    3m53s
staging-sg   v1.23.4    Push   True    21m

[RainbowMango]

Now the expected behavior of unjoin is removing all propagated resources from the target cluster.

During removing a cluster, the Cluster Object will be removed first, that will lead to cleanup propagated resources by cluster-controller.

Unjoin cluster from karmada control plane, but not to remove resources created by karmada in the unjoining cluster karmadactl unjoin CLUSTER_NAME

The example description in karmadactl unjoin is not correct or misleading.

@icy Thank you for spotting.

We can consider supporting the case that keeps propagated resources during unjoin. We might need some more feedback.

I want my cluster to be disjoined but the propagated resources are still there. However, the following command still ends up in deleting all my resources:

The first question is why you are going to unjoin a cluster?

And then, will this cluster be joined in the future? I'm asking because the propagated resource might own control labels that was added by Karmada, we should evaluate if it will block the following registration(join) or if there is any conflict.

[icy]

The first question is why you are going to unjoin a cluster?

And then, will this cluster be joined in the future? I'm asking because the propagated resource might own control labels that was added by Karmada, we should evaluate if it will block the following registration(join) or if there is any conflict.

@RainbowMango Thanks for your reply. I don't have a strong reason to retain that feature. What I was trying to do is to "clone" the existing member, and no, this member probably won't be joined in the future. I think this is helpful for my team workflow where they want to have some dedicated place for testing purpose.

[icy]

@RainbowMango I think of another use case. We need to migrate our workload from old cluster to new cluster. For these clusters, we basically don't need karmada in long term (at least, for now): We just need some easy way to replicate the workload and forget. I'm not sure if karmada is better than velero in this situation. Your advice is highly appreciated.

[RainbowMango]

Luckily we have a similar user case at Building a PaaS Platform with Karmada to Run Containers --VIPKID, they often buy Kubernetes clusters from different cloud providers like AWS, Google Cloud. As the infrastructure team, they will deploy some basic applications(like monitor, logging, platform and so on) before deliver the cluster to other team, they use Karmada to propagate these applications to rapidly deploy a new cluster, and just need to manage a single configuration. But in this case, the cluster won't be enjoined after deploying as they might update the configurations at any time.

What I was trying to do is to "clone" the existing member, and no, this member probably won't be joined in the future.

What's your concern here, that you want to detach the cluster from Karmada?

I'm not sure if karmada is better than velero in this situation. Your advice is highly appreciated.

I'm not an expert of Velero, as far as I know, Velero focuses on backup and restore, and Karmada focuses on manage the configurations across clusters. Velero will monotonously back up configurations from an existing cluster and then restore to another cluster. Note that if the configuration is different between clusters, you might need Karmda OverridePolicy.

[icy]

@RainbowMango thanks again for your time on the feedback.

What's your concern here, that you want to detach the cluster from Karmada?

I'd like to have some feature to ensure that all propagated resources on the new cluster are still available after the cluster is detached from the Karamda cluster. The use case of VIPKID is interesting to me, however I haven't got how they could do that. So far I can't do the same with the command kamardactl unjoin.

I think that Karmada has conflictResolution option to override the existing resources. To me it makes sense to have some option for kamardactl unjoin to retain the resources; if the cluster may join later, the option conflictResolution can help.

I'm not an expert of Velero, as far as I know, Velero focuses on backup and restore, and Karmada focuses on manage the configurations across clusters. Velero will monotonously back up configurations from an existing cluster and then restore to another cluster. Note that if the configuration is different between clusters, you might need Karmda OverridePolicy.

I got some initial read on our documentation: https://karmada.io/docs/administrator/backup/working-with-velero/#backup-and-restore-of-kubernetes-resources-through-velero-combined-with-karmada. Though I understand the point of backup (which is not really cool to me, because if a backup is not tested, it's still not a good backup), the other features confuse me a bit. I think migration/cluster replication can be done exactly well with karmada, which is also the thing I was trying to do and get into this issue.

Btw, sorry if this is quite OT for the team and everyone.

[a7i]

karmada controller-manager (by default) propagates all namespaces, unless we provide a flag to skip a namespace by name or pattern.

We ran into a situation where unjoining a cluster attempted to delete all namespaces, even for those that did not have any resources managed by a policy. This was a surprise to us and I think there is a use-case to unjoin a cluster with a capability to similar to --cascade=orphan. At minimum, in case of resources that are automatically propagated (e.g. Namespaces)