kubeflow
Support Volcano Scheduler in Kubeflow Trainer
What you would like to be added?
In Kubeflow Training Operator V1, we support Volcano for gang-scheduling, while Trainer V2 hasn't supported it yet.
Since Volcano is a widely adopted scheduler for AI workloads, it could provide Trainer with more AI-specific scheduling capabilities if we integrate Volcano into Trainer, thus benefiting users who want to schedule pods with Volcano on top of Kubeflow Trainer.
/cc @kubeflow/wg-training-leads @saileshd1402 @astefanutti @juliusvonkohout @franciscojavierarceo @varodrig @rareddy @thesuperzapper @seanlaii @deepanker13 @helenxie-bit @Doris-xm @truc0 @mahdikhashan
Why is this needed?
In #2182, users requested for richer Volcano support in Kubeflow Training Operator V1.
AFAIK, kubeedge/sedna is waiting for the support of Volcano to enable gang-scheduling in edge-cloud environments: https://github.com/kubeedge/sedna/issues/463. One of the reasons why it was paused is due to:
- All training workers must have the same parameters: The
PyTorchJobCRD in training-operator assumes that all training workers(pods) shared the same training parameters, whileFederatedLearningJobCRD in Sedna allows training workers to have different training parameters. So we assume that all training workers have the same training parameters, which will surely put many restrictions on the applied scenarios of Senda Federated Learning V2 but we have no choice.
In Kubeflow Trainer V2, we introduce jobset as the low-level runtime for distributed training, which allows users to define multiple training parameters for different training workers. It's a good choice to adopt Kubeflow Training V2 instead of the V1 version for them.
Based on the reasons above, supporting Volcano can bring users with great values.
Love this feature?
Give it a 👍 We prioritize the features with most 👍
@andreyvelich I find that there is no label like area/scheduling in issues. Could you please create one and attach it to this issue? Thanks a lot.
Nice feature! I would like to know what changes are there in the trainer v2 compared to v1?
@JesseStutler Hi, thanks for your interest in Kubeflow Trainer.
In v1, we create CRDs for every kind of ML frameworks, like PyTorchJob, TFJob, PaddleJob. And users need to know about Kubernetes to complete the configuration items in the CRD, which is not friendly with data scientists.
In v2, we unify these CRDs into TrainingRuntime and TrainJob. You can simply consider them as:
TrainingRuntime: A blueprint for training, which will be completed by DevOps Engineers and ML Engineers.TrainJob: Do mutation onTrainingRuntime. Data scientists can change the model, dataset and other training-related configurations. This will allow them to quickly iterate their ideas and accelerate AI development.
Also, we provide SDK for data scientists to do the mutation. They do not need to know about Kubernetes any more while leveraging the abilities provided by Kubernetes.
You can find more information in this video and the design doc if you are interested in it.
/cc @kubeflow/wg-training-leads @astefanutti Do you have other complements?
Oh that would be great, just like Kserve ServingRuntime, If there are any volcano communities that need adaptation, please let us know, we are willing to contribute
Thanks! This issue will be converted to a GSoC project this year. Our communities can bond with each other in this summer!
Hi，I am currently working on mlops projects, and I am familiar with kubernetes. I also happen to know about kubeflow and volcano, so this topic seems to be suitable for me to participate in. I hope to have the opportunity to contribute this topic to the kubeflow community.
Thanks! This issue will be converted to a GSoC project this year. Our communities can bond with each other in this summer!
Hi, I'm currently working for AI workload scheduler. Can you assign this to me?
@rudrakshs-cerebras This issue is created for GSoC project. So probably I can't assign it to you. I'm willing to help if you could take another issue with good-first-issue label:)
I’m interested in participating in GSoC via this project. I have a strong understanding of AI workload scheduling and Kubernetes in general, and I believe I can contribute effectively to this feature.
@rudrakshs-cerebras In that case, you can make some contributions to kubeflow and raise your proposal for this project. Following the GSoC guidelines of choosing contributors, we'll review the proposal and take your contributions into consideration to make the final decision. I believe, it will be more fair for other contributors who also want to participate in.
https://github.com/kubeflow/trainer/issues/1851 Starting with this then :) Thanks, will work on the proposal. @Electronic-Waste
Hi @Monokaix @JesseStutler, I'm happy to share with you that this issue has been chosen as one of the GSoC projects for Kubeflow in this year! It's thrilling to see that Kubeflow is going to bond with the Volcano community.
In this summer, @Doris-xm will carry on this project. @rudeigerc and I will serve as the mentor for this project. Let's move forward!
The link for the design doc in the issue comment https://github.com/kubeflow/trainer/issues/2437#issuecomment-2690101139 is broken. I found it already though.
@anindya-saha Thanks for raising this. I've fixed the broken link.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This has been completed: https://github.com/kubeflow/website/pull/4213. Great work @Doris-xm 🎉