Many low prioirty jobs can block a high-priority job with "resource in cluster is overused"

[mattwelborn]

Description

I have two types of jobs:

1. Low-priority jobs. This job uses a priority class with priority 10 and a queue with weight 1. This job is set to tolerate only node 1. (Using either node affinity or taints.)
2. High-priority jobs. This job uses a priority class with priority 100 and a queue with weight 100. This job is set to tolerate only node 2. (Using either node affinity or taints.)

When I submit many low-priority jobs and then submit one high-priority job, the high-priority job does not run. It does not create pods. Its podgroup has an event which says "resource in cluster is overused".

Steps to reproduce the issue

1. Create a k8s cluster with two nodes: node-1 and node-2.
2. Create two priority classes:
   1. pc-low has priority 10.
   2. pc-high has priority 100.
3. Create two queues:
   1. q-low has weight 1.
   2. q-high has weight 100.
4. Create a low priority job definition job-low which uses pc-low, q-low, can only be scheduled onto node-1, and which sleeps for 5 minutes.
5. Create a high priority job definition job-high which uses pc-high, q-high, can only be scheduled onto node-2, and which sleeps forever.
6. Submit 1000 copies of job-low.
7. Submit 1 copy of job-high.

Describe the results you received and expected

Results received: many pods are created for the copies of job-low until there are ~80 pending pods corresponding to job-low. These copies of job-low run on node-1, while node-2 remains idle. The one copy of job-high creates a podgroup, but the podgroup fails to create a pod. Instead, it repeatedly gives the error "resource in cluster is overused". Even as the jobs corresponding to job-low complete, job-high does not run.

Expected results: the copies of job-low should run in order of submission on node-1. (This happens.) The one copy of job-high should immediately run on node-2. (This does not happen.)

What version of Volcano are you using?

v1.9.0

Any other relevant information

I believe that my Volcano scheduler config is the default:

  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
        enablePreemptable: false
      - name: conformance
    - plugins:
      - name: overcommit
      - name: drf
        enablePreemptable: false
      - name: predicates
      - name: proportion
      - name: nodeorder
      - name: binpack