Add job suspend semantics

[xiaoxubeii]

To support job suspend semantics like Kubernetes batch job: https://kubernetes.io/docs/concepts/workloads/controllers/job/#suspending-a-job

[google-oss-prow[bot]]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: To complete the pull request process, please assign gaocegege after the PR has been reviewed. You can assign the PR to them by writing /assign @gaocegege in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[gaocegege]

/ok-to-test

[gaocegege]

Thanks for the PR, is it ready to review?

[xiaoxubeii]

Thanks for the PR, is it ready to review?

@gaocegege Ready for review. Thanks :)

[ggaaooppeenngg]

How is this PR going now?

[alculquicondor]

Is this actively being worked on? Or will we get rid of the common repo first?

[tenzen-y]

Is this actively being worked on? Or will we get rid of the common repo first?

@alculquicondor Maybe, we will work on the Job suspend feature in the next kubeflow release cycle (maybe kubeflow v1.8?). Since we didn't push this feature to the enhancement lists for the next kubeflow release (v1.7) and the feature freeze for the next kubeflow version (v1.7) is coming up.

https://github.com/kubeflow/training-operator/issues/1683

Wed Jan 25th 2023 Week 18 Release Team Feature Freeze

https://github.com/kubeflow/community/blob/6ba2e0e754166989d2f0d06aae827ceafdb65b29/releases/release-1.7/README.md

[johnugeorge]

Agree. we will take this up in next release after our merging kubeflow/common as planned in https://github.com/kubeflow/training-operator/issues/1714#issuecomment-1374537434

[alculquicondor]

@tenzen-y how do you feel about starting with the integration for mpi-operator v2 and follow through with training-operator later? It might give us a better chance to iterate faster and learn.

[tenzen-y]

@tenzen-y how do you feel about starting with the integration for mpi-operator v2 and follow through with training-operator later? It might give us a better chance to iterate faster and learn.

@alculquicondor Yes. that is a good idea. I was thinking of the same. Although, we need to move forward https://github.com/kubernetes-sigs/kueue/issues/369 before we adapt mpi-operator to Kueue.

[alculquicondor]

Excellent! We can work on the kueue side in parallel, while we add support for suspend in the mpi-operator.

[tenzen-y]

Excellent! We can work on the kueue side in parallel, while we add support for suspend in the mpi-operator.

You are right. I will work on the following steps after kubeflow feature freeze date (1/25) since I have no enough bandwidth for mpi-operator v2, now:

1. https://github.com/kubeflow/mpi-operator/pull/502
2. https://github.com/kubeflow/mpi-operator/issues/500
3. Support suspend in mpi-operator

Although, other anyone can take step 3 after step 1 is completed.

[alculquicondor]

@mimowo will help with suspend in mpi-operator https://github.com/kubeflow/mpi-operator/issues/504

[tenzen-y]

Great! Thanks to @mimowo!

[xiaoxubeii]

Is this actively being worked on? Or will we get rid of the common repo first?

@alculquicondor Maybe, we will work on the Job suspend feature in the next kubeflow release cycle (maybe kubeflow v1.8?). Since we didn't push this feature to the enhancement lists for the next kubeflow release (v1.7) and the feature freeze for the next kubeflow version (v1.7) is coming up.

kubeflow/training-operator#1683

Wed Jan 25th 2023 Week 18 Release Team Feature Freeze

https://github.com/kubeflow/community/blob/6ba2e0e754166989d2f0d06aae827ceafdb65b29/releases/release-1.7/README.md

Agreed. We could try to work on Job suspend feature for kubeflow v1.8.

[alculquicondor]

@johnugeorge how are we doing with the branch creation? Can we proceed with this PR or move it to training-operator?