fix(runtimes): propagate Trainer.NumNodes into TemplateSpec (Parallelism/Completions)
What type of PR is this?
/kind bug
What this PR does / why we need it:
This PR fixes the RuntimeInfo helpers to correctly synchronize PodSets.Count values (set by numNodes) to the TemplateSpec.ObjApply (JobSet specification).
Previously, when external consumers like Kueue called RuntimeInfo(), the info.TemplateSpec.PodSets[*].Count was correctly updated by ML policy plugins (Torch, PlainML, MPI), but the underlying JobSet spec (info.TemplateSpec.ObjApply) still had the original Parallelism and Completions values. This required consumers to implement post-processing workarounds.
Changes:
- Added
SyncPodSetsToTemplateSpec()method inruntime.gothat syncsPodSets.Countto JobSet'sParallelism/Completions - Called this method in
RuntimeInfo()afterRunEnforceMLPolicyPlugins()runs - Added unit tests for the new sync function
Which issue(s) this PR fixes:
Fixes https://github.com/kubeflow/trainer/issues/3042
Related Kueue issue: https://github.com/kubernetes-sigs/kueue/issues/8287
Special notes for your reviewer:
- The existing comment in
runtime.goline 45 already mentioned: "ObjApply podSpecs and this PodSets should be kept in sync by info.SyncPodSetsToTemplateSpec()" - this PR implements that function. - Once this is merged, the workaround in Kueue's
trainjob_controller.gocan be removed. - All existing tests pass.
[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign jeffwan for approval. For more information see the Kubernetes Code Review Process.
The full list of commands accepted by this bot can be found here.
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
🎉 Welcome to the Kubeflow Trainer! 🎉
Thanks for opening your first PR! We're happy to have you as part of our community 🚀
Here's what happens next:
- If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.
- Our team will review your PR soon! cc @kubeflow/kubeflow-trainer-team
Join the community:
- Slack: Join our #kubeflow-trainer Slack channel.
- Meetings: Attend the Kubeflow AutoML and Training Working Group bi-weekly meetings.
Feel free to ask questions in the comments if you need any help or clarification! Thanks again for contributing to Kubeflow! 🙏
/ok-to-test