volcano icon indicating copy to clipboard operation
volcano copied to clipboard

Scheduler supports outputting more detailed scheduling events

Open Poor12 opened this issue 11 months ago • 1 comments

We use volcano to schedule Pods in production environments. Recently, we found that it is difficult to directly locate the cause when Pod scheduling fails, and R&D personnel often need to use the scheduler log to assist in locating the cause.

For example,we use an under-resourced queue to schedule Pods. When we describe pod events, we only can see podgroup is not ready. However, podgroup is like an internal object of volcano. We don't want our operations staff to be aware of it.

Events:
  Type     Reason            Age    From     Message
  ----     ------            ----   ----     -------
  Warning  FailedScheduling  4m31s  volcano  pod group is not ready, 1 Pending, 1 minAvailable; Pending: 1 Unschedulable

Even if we drill down into the PodGroup, sometimes we still can't quickly figure out the reason. For example, we can see that queue resource quota is insufficient, but we do not know specific resource. We can see that queue is overused, we also do not known specific resource. This is not friendly to operations.

Events:
  Type     Reason         Age                     From     Message
  ----     ------         ----                    ----     -------
  Warning  Unschedulable  6m18s (x12 over 6m29s)  volcano  1/1 tasks in gang unschedulable: pod group is not ready, 1 Pending, 1 minAvailable; Pending: 1 Unschedulable
  Warning  Unschedulable  6m18s (x12 over 6m29s)  volcano  1/1 tasks in gang unschedulable: pod group is not ready, 1 Pending, 1 minAvailable; Pending: 1 Unschedulable
  Warning  Unschedulable  6m17s (x12 over 6m28s)  volcano  1/1 tasks in gang unschedulable: pod group is not ready, 1 Pending, 1 minAvailable; Pending: 1 Unschedulable
  Normal   Unschedulable  89s (x297 over 6m29s)   volcano  queue resource quota insufficient
  Normal   Unschedulable  89s (x297 over 6m29s)   volcano  queue resource quota insufficient
  Normal   Unschedulable  89s (x297 over 6m29s)   volcano  queue resource quota insufficient

Poor12 avatar May 22 '25 12:05 Poor12

what is the version of scheduler?

lowang-bh avatar Jun 01 '25 14:06 lowang-bh

what is the version of scheduler?

Sorry for late reply. I'm using release-1.11.

Poor12 avatar Jul 13 '25 01:07 Poor12