Integration into kubeflow pipeline

[miguelvr]

I'm completely new to kubeflow, but the main advantage of it from my perspective is the usage of pipelines to setup production ready end to end machine learning workflows.

I'm trying to port production code that we have into a kubeflow pipeline. However, one of the steps is to perform distributed training. I understand how to set up a pipeline that uses a small number of GPUs within the same pod, but still don't get how to integrate distributed training with a PyTorchJob into a kubeflow pipeline.

Is this possible? What are the current solutions for this?

[johnugeorge]

We don't have it added into pipeline yet

[lainisourgod]

@johnugeorge Hi! Do you have any roadmap about integrating into pipelines?

Looking at TFJob pipeline integration it seems not to be hard.

[jtfogarty]

/kind question /area pipelines /priority p2

[k8s-ci-robot]

@jtfogarty: The label(s) area/pipelines cannot be applied, because the repository doesn't have them

In response to this:

/kind question /area pipelines /priority p2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jtfogarty]

/area engprod

[ca-scribner]

I've seen discussion on launchers here and on Kubeflow/Pipelines, but nothing concrete. Has anyone built a solution for this? If not, I plan to create one based on the TFJob launcher and can share

[gaocegege]

@ca-scribner Thanks! I do not think there is progress from our side.