Custom driver support for KubeflowV2DagRunner

[jeongukjae]

System information

• TFX Version (you are using): 1.14
• Environment in which you plan to use the feature (e.g., Local (Linux/MacOS/Windows), Interactive Notebook, Google Cloud, etc..): Vertex AI
• Are you willing to contribute it (Yes/No): yes

Describe the feature and the current behavior/state.

Currently, KubeflowV2DagRunner does not support custom driver class. It only supports file based example gen driver which is located in

https://github.com/tensorflow/tfx/blob/master/tfx/orchestration/kubeflow/v2/file_based_example_gen/driver.py

Will this change the current API? How?

Not expected to change the API. But this will change following codes.

• https://github.com/tensorflow/tfx/tree/master/tfx/orchestration/kubeflow/v2/file_based_example_gen
• https://github.com/tensorflow/tfx/blob/master/tfx/orchestration/kubeflow/v2/step_builder.py

Who will benefit with this feature?

Developers who want to leverage the tfx concepts more widely.

Do you have a workaround or are completely blocked by this? :

Blocked by this now.

Name of your Organization (Optional)

Any Other info.

[singhniraj08]

@jeongukjae, Since you mentioned you are willing to contribute for this feature, I will request you to go ahead a create a PR for feature implementation. Thank you for the contributions.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[jeongukjae]

do not close this

[pritamdodeja]

@jeongukjae can you please describe what you're trying to accomplish? I have successfully implemented components which are at the beginning of the pipeline that didn't use file based example gen. To your point about KubeflowV2DagRunner, is your pipeline executable by LocalDagRunner? It's possible there's some translation issue related to ML Metadata etc., let me know what your situation is, and we can figure out how to solve it.