feature: unjoin a cluster gracefully

[Garrybest]

What would you like to be added: Now when using karmadactl unjoin, we delete the cluster directly. The scheduling result in resource binding still remains. I would like to make unjoin more gracefully. The procedure could refer to what kubectl drain does.

1. Cordon the target cluster.
2. Delete the scheduling result in RB/CRB which has been scheduled to the target cluster.
3. Delete the cluster.

Why is this needed: Make unjoin a cluster more gracefully.

[Garrybest]

/cc @lonelyCZ

[Garrybest]

Related to #1762.

[lonelyCZ]

Delete the scheduling result in RB/CRB which has been scheduled to the target cluster.

Should we execute all these actions in cli tool? Or is there a controller to clear the scheduling result in RB/CRB?

[Garrybest]

I think so, or we may add a NoExecute taint in this cluster and wait for the eviction. Which one do you prefer?

FYI, in k/k, kubectl drain delete pods in CLI tool.

[lonelyCZ]

FYI, in k/k, kubectl drain delete pods in CLI tool.

Looks good, I am going to research it.

[carlory]

@lonelyCZ @Garrybest Can I pick up it ?

[lonelyCZ]

@carlory Sure, thanks for your interest to it., you can /assign to youself. Look forward to your contribution!

[Garrybest]

/help /assign @carlory

Thanks, glad to hear that. FYI, https://github.com/kubernetes/kubernetes/blob/050f930f8968874855eb215f0c0f0877bcdaa0e8/staging/src/k8s.io/kubectl/pkg/cmd/drain/drain.go#L293-L334

[karmada-bot]

@Garrybest: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to this:

/help /assign @carlory

Thanks, glad to hear that. FYI, https://github.com/kubernetes/kubernetes/blob/050f930f8968874855eb215f0c0f0877bcdaa0e8/staging/src/k8s.io/kubectl/pkg/cmd/drain/drain.go#L293-L334

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Garrybest]

Hi @carlory, any progress here?

[carlory]

So sorry, I forget it.

I will do it this week, is still ok ?

@Garrybest

[Garrybest]

Thanks @carlory, take your time😄

[RainbowMango]

This is another scenario of eviction. Do you know where can I find the PDB/evict-sub-resource thing? share a link? @Garrybest

[Garrybest]

Eviction is a subresource of Pod.

https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets

Cluster managers and hosting providers should use tools which respect PodDisruptionBudgets by calling the Eviction API instead of directly deleting pods or deployments.

[RainbowMango]

Is there a solution for this issue?

I'm thinking if it's the responsibility of the karmada-scheduler to remove the legacy schedule results?

[Garrybest]

I don't think it's scheduler's duty. karmadactl unjoin is like what kubectl drain does:

1. Cordon the target cluster.
2. Delete the scheduling result in RB/CRB which has been scheduled to the target cluster.
3. Delete the cluster.

[RainbowMango]

Do you mean you want CLI karmadactl unjoin to do the cleanup work?

I suppose CLI is a kind of one-off job, it can't guarantees to finish all the cleanup work in a single request, and due to probably huge amount of RB/CRB, it might increase the time that user don't need to care.

So, my opinion is to put the cleanup job to the controller-manager(maybe cluster-controller?) (ignore scheduler, let it focus on schedule).

[Garrybest]

I have talked with @RainbowMango privately. We now think it's better to do the cleanup work in the cluster_controller here before the finalizer is removed instead of in CLI tools.

Now the plan is changed. Hi @carlory, would you still like to make a contribution here? It's up to you.

[carlory]

I am willing to try, but I need some time to learn how to implement it.

[Garrybest]

fix in #2373

[RainbowMango]

/assign @Garrybest