MVP implementation of Hierarchical Proportion

[hzq5477]

issue 3018

Due to API changes, this pr needs to be merged first

This PR is for MVP implementation of the hierarchical proportion plugin, where I

• implemented the data structure of hierarchical queues
• built the tree when a session opens
• add isLeaf Function to determine whether the queue is a leaf node
• add updateParentQueue to update the resource state of parent queues based on the child queues' states.

modified:

• AddQueueOrderFn(): sorts queues for resource allocation, prioritizing leaf queues before non-leaf queues.
• AddJobEnqueueableFn(): rejects jobs if not in a leaf queue.
• AddAllocatableFn(): return false jobs if not in a leaf queue.
• AddOverusedFn(): when a leaf queue is overusing, it is possible that its parent is also overusing resources, add check if parent queue is overusing in this function

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: To complete the pull request process, please assign thor-wl You can assign the PR to them by writing /assign @thor-wl in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[Monokaix]

Seems that deserved resource computation and reclaim involveing hierarchical queue is not considered in current implement, we'd better clarify the difference of hierarchical and non-hierarchical scenario first: ）

[noobzzw]

Hi, this is a remarkable feature. If we can implement hierarchical queues in Volcano, it can accelerate the replacement of the YARN scheduler in big data scenarios. The current pull request has not been updated for a while, and I have implemented a similar logic in my work. Could we break down some sub-tasks and allow me to contribute as well :)

[Monokaix]

Hi, this is a remarkable feature. If we can implement hierarchical queues in Volcano, it can accelerate the replacement of the YARN scheduler in big data scenarios. The current pull request has not been updated for a while, and I have implemented a similar logic in my work. Could we break down some sub-tasks and allow me to contribute as well :)

That's great! You can keep working on this, and we have an original design doc here https://github.com/volcano-sh/volcano/pull/3215, and you can implement based on that, if you have any question welcome discuss here.

[noobzzw]

Hi, this is a remarkable feature. If we can implement hierarchical queues in Volcano, it can accelerate the replacement of the YARN scheduler in big data scenarios. The current pull request has not been updated for a while, and I have implemented a similar logic in my work. Could we break down some sub-tasks and allow me to contribute as well :)

That's great! You can keep working on this, and we have an original design doc here #3215, and you can implement based on that, if you have any question welcome discuss here.

Hi, thank you very much for your reply. I will continue working on this feature.

[volcano-sh-bot]

@hzq5477: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Monokaix]

Hi, this is a remarkable feature. If we can implement hierarchical queues in Volcano, it can accelerate the replacement of the YARN scheduler in big data scenarios. The current pull request has not been updated for a while, and I have implemented a similar logic in my work. Could we break down some sub-tasks and allow me to contribute as well :)

That's great! You can keep working on this, and we have an original design doc here #3215, and you can implement based on that, if you have any question welcome discuss here.

Hi, thank you very much for your reply. I will continue working on this feature.

Hi, any updates here? Hope for your reply: )

[noobzzw]

Hi, this is a remarkable feature. If we can implement hierarchical queues in Volcano, it can accelerate the replacement of the YARN scheduler in big data scenarios. The current pull request has not been updated for a while, and I have implemented a similar logic in my work. Could we break down some sub-tasks and allow me to contribute as well :)

That's great! You can keep working on this, and we have an original design doc here #3215, and you can implement based on that, if you have any question welcome discuss here.

Hi, thank you very much for your reply. I will continue working on this feature.

Hi, any updates here? Hope for your reply: )

Sorry, I have been busy with my primary job recently and haven't had the chance to organize the PR. However, the feature has already been deployed and running smoothly in my department. I will strive to complete the PR by the end of April : )

[Monokaix]

Hi, this is a remarkable feature. If we can implement hierarchical queues in Volcano, it can accelerate the replacement of the YARN scheduler in big data scenarios. The current pull request has not been updated for a while, and I have implemented a similar logic in my work. Could we break down some sub-tasks and allow me to contribute as well :)

That's great! You can keep working on this, and we have an original design doc here #3215, and you can implement based on that, if you have any question welcome discuss here.

Hi, thank you very much for your reply. I will continue working on this feature.

Hi, any updates here? Hope for your reply: )

Sorry, I have been busy with my primary job recently and haven't had the chance to organize the PR. However, the feature has already been deployed and running smoothly in my department. I will strive to complete the PR by the end of April : )

Great to hear that~ Well done！

[Monokaix]

Hi, this is a remarkable feature. If we can implement hierarchical queues in Volcano, it can accelerate the replacement of the YARN scheduler in big data scenarios. The current pull request has not been updated for a while, and I have implemented a similar logic in my work. Could we break down some sub-tasks and allow me to contribute as well :)

That's great! You can keep working on this, and we have an original design doc here #3215, and you can implement based on that, if you have any question welcome discuss here.

Hi, thank you very much for your reply. I will continue working on this feature.

Hi, any updates here? Hope for your reply: )

Sorry, I have been busy with my primary job recently and haven't had the chance to organize the PR. However, the feature has already been deployed and running smoothly in my department. I will strive to complete the PR by the end of April : )

Hi, is any progress here?

[noobzzw]

Hi, this is a remarkable feature. If we can implement hierarchical queues in Volcano, it can accelerate the replacement of the YARN scheduler in big data scenarios. The current pull request has not been updated for a while, and I have implemented a similar logic in my work. Could we break down some sub-tasks and allow me to contribute as well :)

That's great! You can keep working on this, and we have an original design doc here #3215, and you can implement based on that, if you have any question welcome discuss here.

Hi, thank you very much for your reply. I will continue working on this feature.

Hi, any updates here? Hope for your reply: )

Sorry, I have been busy with my primary job recently and haven't had the chance to organize the PR. However, the feature has already been deployed and running smoothly in my department. I will strive to complete the PR by the end of April : )

Hi, is any progress here?

Hi, I'm very sorry. Due to work and personal reasons, I haven't been able to submit the relevant code for merge request in the past 3 months. However, I still want to contribute to this feature. By the way, I have a few questions that I was hoping you could help me with:

1. I noticed that volcano has decoupled the capacity schedule from the proportion plugin, which is a great decision. Shouldn't the hierarchical queue be implemented in the capacity plugin instead?
2. I noticed that this feature has been registered for Open Source Summer: https://summer-ospp.ac.cn/org/prodetail/243ba0509?list=org&navpage=org, and that the general idea is similar to mine. Can I still submit the code? Or should I wait for the corresponding pr and do some optimization work based on it?

[Monokaix]

Hi, this is a remarkable feature. If we can implement hierarchical queues in Volcano, it can accelerate the replacement of the YARN scheduler in big data scenarios. The current pull request has not been updated for a while, and I have implemented a similar logic in my work. Could we break down some sub-tasks and allow me to contribute as well :)

That's great! You can keep working on this, and we have an original design doc here #3215, and you can implement based on that, if you have any question welcome discuss here.

Hi, thank you very much for your reply. I will continue working on this feature.

Hi, any updates here? Hope for your reply: )

Sorry, I have been busy with my primary job recently and haven't had the chance to organize the PR. However, the feature has already been deployed and running smoothly in my department. I will strive to complete the PR by the end of April : )

Hi, is any progress here?

Hi, I'm very sorry. Due to work and personal reasons, I haven't been able to submit the relevant code for merge request in the past 3 months. However, I still want to contribute to this feature. By the way, I have a few questions that I was hoping you could help me with:

1. I noticed that volcano has decoupled the capacity schedule from the proportion plugin, which is a great decision. Shouldn't the hierarchical queue be implemented in the capacity plugin instead?
2. I noticed that this feature has been registered for Open Source Summer: https://summer-ospp.ac.cn/org/prodetail/243ba0509?list=org&navpage=org, and that the general idea is similar to mine. Can I still submit the code? Or should I wait for the corresponding pr and do some optimization work based on it?

You're right, but I think there is no conflict between these two quue plugin, you can keep working on this: )