karmada
karmada copied to clipboard
karmada-io
Discussion about adding propagate priority during scheduling
What would you like to be added:
Add a new field PropagatePriority into ClusterAffinity so that user could have a propagating preference with multiple levels of priority.
User Story:
- Story 1
Image that we have our own private cloud clusters which are managed by karmada. When the load of clusters reaches the peak, we must scale our workload to public cloud clusters because the resources in our private cloud is limited.
However, the cost in public clusters are much larger than private clusters. We would like to propagate replicas as many as possible between our private clusters, the rest ones are going to be propagated between public clusters.
- https://github.com/karmada-io/karmada/issues/2085
Why is this needed:
It makes multi-cluster scheduling more flexible. Clusters are descending sorted by weight and scheduler will try to propagate to the preferred clusters under the premise of constraints, e.g., SpreadConstraints, AvailableReplicas, etc.
/cc @kevin-wangzefeng @RainbowMango @qianjun1993 @mrlihanbo @XiShanYongYe-Chang @gf457832386
@Garrybest Is there a use case that can indicate the benefits to Karmada users?
Do you propose both preferred and required affinity? Or, just preferred? (I can see both from the issue title, but just one from #842)
It is proposed long time ago. I have updated this issue description, please take a look.
Thanks, I see it. it makes sense(just looked at story 1).
This feature belongs to the scope of dividing replicas, I'm not sure if we can put it into the ReplicaSchedulingStrategy. Just a second thought, I'll look at #2085.
I think it's not just for dividing replicas. Let me explain this field.
When propagating, we have some constraints, e.g., spreadConstraints, availableReplicas, etc. What we are going to do is try to find a solution under the constraints.
PropagatePriority aims to tell us in which set we could look for solutions. Imagine we get a priority list with set A, set B and set C by descending order.
- We try to find solutions in set A under the
constraints. - If failed, maybe due to
spreadConstraintsoravailableReplicas, we will try to propagate between set A+B. Note that dividing will have a preference with set A. - Once the
constraintsis met, we could return the solution.
Have a questison, why not define the PropagatePriority in ReplicaScheduling?
If define in ClusterAffinity, if it's static weight scheduling strategy, the priority will works?
@Garrybest
Have a questison, why not define the PropagatePriority in ReplicaScheduling?
I suppose this field is a kind of ClusterAffinity, matching the semantics.
If define in ClusterAffinity, if it's static weight scheduling strategy, the priority will works?
No, it only works for dynamic scheduling.
It must be the file of ClusterAffinity(Will it make customers confused because of scheduling strategy)? I think the only benifit is it can use label selector to give a priority for multiple clusters.
It must be the file of ClusterAffinity(Will it make customers confused because of scheduling strategy)?
Well, it could be, the concern is sensible. This field is actually a kind of soft affinity when dividing replicas, and does not work if policy is static or dupliacated. I'm ok of moving to ReplicaScheduling .
I think the only benifit is it can use label selector to give a priority for multiple clusters.
No worries. Label selector also works as well no matter where this field is located, corev1.PreferredSchedulingTerm could prescribe affinity terms.
BTW, if define it in ReplicaScheduling, but without any labelSelector, users need to set each clusers' priority one by one. will it make configuration complex? or we should give this configuration way also?
Users need to set each clusers' priority one by one.
I think there is a misjudge here. According to https://github.com/karmada-io/karmada/pull/842, priority is a slice []corev1.PreferredSchedulingTerm. No matter the location is, users always need to set a preference which consists of matchExpressions and matchFields. Label or cluster name is both available here.
If the scheduling policy is considered, the following scenarios are involved:
| clusters' priority is configured | clusters' priority is not configured | user story | |
|---|---|---|---|
| Duplicated | Behaving the same way now | Behaving the same way now | |
| Divided/StaticWeight | If the weights are the same, high-priority clusters are preferentially scheduled(Very few benefits) | Behaving the same way now | |
| Divided/DynamicWeight | If the weights are the same, high-priority clusters are preferentially scheduled(Very few benefits) | Behaving the same way now | |
| Divided/Aggregated | High-priority clusters are preferentially selected for initial scheduling and scaling up/Low-priority clusters are preferentially selected for scaling down. | Behaving the same way now | Preferentially schedule to the local clusters(IDC) |
cc @Garrybest, what do u think?
Generally accurate, expect DynamicWeight. As long as the strategy is dynamic, i.e. DynamicWeight or Aggregated, cluster priority will take effect.
If DynamicWeight, users also want to distinguish different kinds of clusters, like public cloud, private cloud or something else.
@Garrybest I added an agenda to today's meeting hoping to share the draft proposal there, meet you there?
@RainbowMango: Closing this issue.
In response to this:
/close
This issue is tracked by #3126 and #3153.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
