volcano-sh
Fix Issue 2262: add priority capability for reclaim action
Fixes #2262
- priority plugin support
reclaimableswitch and add UT about priority plugin - priority plugin docs
Note: reclaimableFn usually is used in reclaim action to reclaim a queue's deserved resource when cluster has not enough resource to allocate new coming tasks in this queue. So please be carefully to set enableReclaimable to true in priority plugin, in case that a queue's resource owned by high priority jobs can not be released. And enableReclaimable is disabled by default for compatibility
If other queue's jobs all have higher priority，current queue can not reclaim their resources, reclaim will not happen, is this reasonable？
If other queue's jobs all have higher priority，current queue can not reclaim their resources, reclaim will not happen, is this reasonable？
We just support this feature and close it by default。 How to use it is depended on the cluster admin.
One solutions is to limit the higher priority jobs used resource not exceed queue's deserved in application layer.
If other queue's jobs all have higher priority，current queue can not reclaim their resources, reclaim will not happen, is this reasonable？
We just support this feature and close it by default。 How to use it is depended on the cluster admin.
One solutions is to limit the higher priority jobs used resource not exceed queue's deserved in application layer.
But this seems places high demands on administrators and limits the job priority of the queue, we'd better add a desige doc and give some user-guide.
[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign william-wang
You can assign the PR to them by writing /assign @william-wang in a comment when ready.
The full list of commands accepted by this bot can be found here.
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
we'd better add a desige doc and give some user-guide.
@Monokaix docs is added.
cc @wangyang0616 @hwdef please help to review. thanks.
@lowang-bh: Closed this PR.
In response to this:
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
@lowang-bh: Reopened this PR.
In response to this:
/reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
It conflicts with the existing allocation logic between queues. When we allocate tasks, we do not consider the priority of jobs between queues, but only consider the priority at the queue level.
I think there may be a bug in Reclaimable in session_plugins.go, for this configmap
tiers:
- plugins:
- name: priority
- name: proportion
as reclaim enabled in priority plugin, let us image that theres some victims return from priority, but if no victims return from proportion plugin , Reclaimable function will return no victims, and reclaim not work any more, put proportion in front of priority can fix this, but it may be a bug? @hwdef
I think there may be a bug in Reclaimable in session_plugins.go, for this configmap
tiers: - plugins: - name: priority - name: proportionas reclaim enabled in priority plugin, let us image that theres some victims return from priority, but if no victims return from proportion plugin , Reclaimable function will return no victims, and reclaim not work any more, put proportion in front of priority can fix this, but it may be a bug? @hwdef
That is a problem of your config. You'd better to put plugins about resource in a same tier, but a tier different from gang/priority, eg:
tiers:
- plugins:
- name: priority
enableReclaimable: false
- name: gang
enablePreemptable: false
- name: conformance
- plugins:
- name: overcommit
- name: drf
enablePreemptable: false
- name: predicates
- name: proportion
- name: nodeorder
- name: binpack
I think there may be a bug in Reclaimable in session_plugins.go, for this configmap
tiers: - plugins: - name: priority - name: proportionas reclaim enabled in priority plugin, let us image that theres some victims return from priority, but if no victims return from proportion plugin , Reclaimable function will return no victims, and reclaim not work any more, put proportion in front of priority can fix this, but it may be a bug? @hwdef
That is a problem of your config. You'd better to put plugins about resource in a same tier, but a tier different from gang/priority, eg:
tiers: - plugins: - name: priority enableReclaimable: false - name: gang enablePreemptable: false - name: conformance - plugins: - name: overcommit - name: drf enablePreemptable: false - name: predicates - name: proportion - name: nodeorder - name: binpack
ok, make sense
@lowang-bh , there is another promble as following:
actions: "enqueue,allocate,backfill,preempt,reclaim"
tiers:
- plugins:
- name: priority
- name: gang
enablePreemptable: false
enableJobStarving: false
enableReclaimable: false
enabledQueueScoreOrder: false
- name: conformance
- plugins:
- name: predicates
- name: proportion
gang enablePreemptable: false( make it true also don't work too), so it's cant do any preempt/reclaim, when a high priority job comes, if there no enough resource to meet gang constraint, I hope get resources by reclaim, but as not meet gang constraint, this job will pending, and reclaim action is just skip pending job, so reclaim not happend, no reclaim, no release resource to get job running, like deadlock, delete enqueue may work, but in my scene，enqueue is a must, and put reclaim in front of enqueue is not good ideal. and overcommit won't use for me. what better soluation to fix it?
@zhoushuke There are two kind of evicts:
- evicts in same queue: this is for high priority job to evict low priority jobs when your queue's deserved resource is used up
- evicts between different queues: a queue's coming job to reclaim their queue's deserved resource from other overused queue. In this scene, the overused resources in other queues are borrowed from this queue, and resource should be give back when need.
If you have any problems about how to use volcano, please file a issue to describe it.
I have some volcano practice and use it in a production to support about 30k~50k pods a day. I think you don't get my point, anyway, I will try to realize it by my own.
enablePreemptable: false
So can you explain why set enablePreemptable: false of gang？
If other queue's jobs all have higher priority，current queue can not reclaim their resources, reclaim will not happen, is this reasonable？
We just support this feature and close it by default。 How to use it is depended on the cluster admin.
One solutions is to limit the higher priority jobs used resource not exceed queue's deserved in application layer.
what about reclaim lower priority first, if no enough resourecs reclaimed then reclaim higher priority task?
this job will pending, and reclaim action is just skip pending job
If you didn't use enqueue action, the default status will be pending and allocate action will change it to enqueue when there is no enqueue action configured. So you should put reclaim action after allocate.
We should consider higher priority jobs in other queues can never be reclaimed if enabledReclaimable is enabled in Priority Plugin. The original design intention of reclaim should be to be able to reclaim jobs in other queues, without considering the priority, right?
