volcano icon indicating copy to clipboard operation
volcano copied to clipboard

NodeSelector not filtering nodes correctly in volcano-scheduler v1.8.2+

Open 1wolfalone1 opened this issue 1 year ago • 5 comments

What happened: I attempted to configure multiple schedulers in Volcano and encountered unexpected behavior with different versions of the volcano-scheduler image.

When I set the volcano-scheduler image to version 1.8.2 (or the default latest in the Helm deployment), image I configured a nodeSelector with volcano.sh/nodegroup-name=g2, which should match two nodes: agent2 and agent3. image image However, the Volcano scheduler logs indicated that it was collecting all nodes in the cluster instead of just those matching the configured node selector. image

This led me to believe there might be an error in my configuration. However, downgrading the volcano-scheduler image to version 1.8.1 or below (within the Helm deployment) resolved the issue. image The logs confirmed this behavior, showing that only nodes agent2 and agent3 were collected, and resource calculation for the scheduler in proportion plugins functioned correctly. image What you expected to happen: The ability to create multiple schedulers with a nodeSelector should function correctly from version 1.8.2 of the volcano-scheduler image onwards.

Environment:

  • Volcano Version: 1.9.0
  • Kubernetes version (use kubectl version): 1.30.1
  • Cloud provider or hardware configuration: k3s and ubuntu 20 vm
  • OS (e.g. from /etc/os-release): ubuntu
  • Install tools: helm

1wolfalone1 avatar Jun 09 '24 03:06 1wolfalone1

Did you also use nodegroup plugin or just multi schedulers?

Monokaix avatar Jun 14 '24 06:06 Monokaix

As a test, I'm currently using multi-schedulers with a node selector in that example. If this approach proves successful, I'll then activate the node group later.

1wolfalone1 avatar Jun 16 '24 21:06 1wolfalone1

There is truly a probelm in versiob v1.8.2,same issue https://github.com/volcano-sh/volcano/issues/3444 , will fix it later: )

Monokaix avatar Jun 20 '24 09:06 Monokaix

/good-first-issue

Monokaix avatar Jun 20 '24 09:06 Monokaix

@Monokaix: This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-good-first-issue command.

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

volcano-sh-bot avatar Jun 20 '24 09:06 volcano-sh-bot

I am interested in this issue, can you assign it to me?

Prepmachine4 avatar Jul 27 '24 15:07 Prepmachine4

I am interested in this issue, can you assign it to me?

Thanks for your warming! It has been fixed by https://github.com/volcano-sh/volcano/pull/3594,you can still select other good first issue to contribute: )

Monokaix avatar Jul 27 '24 15:07 Monokaix

/close

Monokaix avatar Jul 28 '24 01:07 Monokaix

@Monokaix: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

volcano-sh-bot avatar Jul 28 '24 01:07 volcano-sh-bot