training-operator

training-operator copied to clipboard

We noticed that the MPIJob used in the Kubeflow community documentation is v2beta1 version (https://www.kubeflow.org/docs/components/training/mpi/), but it seems that the training-operator only supports the v1 version of MPIJob. Does the training-operator community have plans to support v2?

Currently, if users need to use both MPIJob and PytorchJob together, they need to install mpi-operator to support v2beta1 MPIJob first, and then install training-operator to use the v1 PytorchJob.

@chenxi-seu Yes, we have a plan to support MPIJob v2. Please see https://github.com/kubeflow/training-operator/issues/1906.

@tenzen-y Thank you for your response. Could you please confirm if my current approach is correct? I plan to first install mpi-operator and then install training-operator by configuring --enable-scheme. I only enable PytorchJob to avoid any conflict between the two operators regarding different versions of MPIJob.

@tenzen-y Thank you for your response. Could you please confirm if my current approach is correct? I plan to first install mpi-operator and then install training-operator by configuring --enable-scheme. I only enable PytorchJob to avoid any conflict between the two operators regarding different versions of MPIJob.

Yes, you're right.

@tenzen-y Thank you for your response. Could you please confirm if my current approach is correct? I plan to first install mpi-operator and then install training-operator by configuring --enable-scheme. I only enable PytorchJob to avoid any conflict between the two operators regarding different versions of MPIJob.

Sorry, I have the same problem now, but I don't understand how you did this step.

@tenzen-y Thank you for your response. Could you please confirm if my current approach is correct? I plan to first install mpi-operator and then install training-operator by configuring --enable-scheme. I only enable PytorchJob to avoid any conflict between the two operators regarding different versions of MPIJob.

Sorry, I have the same problem now, but I don't understand how you did this step.

@mupeifeiyi This would be a good example: https://github.com/kubeflow/training-operator/issues/1777#issuecomment-1480720233

/> > > @tenzen-y Thank you for your response. Could you please confirm if my current approach is correct? I plan to first install mpi-operator and then install training-operator by configuring --enable-scheme. I only enable PytorchJob to avoid any conflict between the two operators regarding different versions of MPIJob.

Sorry, I have the same problem now, but I don't understand how you did this step.

@mupeifeiyi This would be a good example: #1777 (comment)

Thanks, the following is useful to me:

    spec:
      containers:
      - args:
        - --enable-scheme=tfjob
        - --enable-scheme=pytorchjob
        - --enable-scheme=mxjob
        - --enable-scheme=xgboostjob
        - --enable-scheme=paddlejob
        command:
        - /manager

mxjob，not mxnetjob

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

For the context here, we are planning to implement support for MPIJob V2 as part of Kubeflow Training V2 proposal: https://bit.ly/3WzjTlw

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@andreyvelich Should we keep this open?

@terrytangyuan We track the V2 migration for MPI jobs as part of this issue: https://github.com/kubeflow/training-operator/issues/2217. In Kubeflow V2 API we will support the second version of MPI operator.

Got it. Thanks