Kubeflow component integration with ML Metadata

[richardsliu]

/kind feature

Why you need this feature: Kubeflow currently doesn't have a unified metadata/artifact management story beyond what's supported in KFP. For example, the concept of a "ML experiment" exists in training and hyperparameter tuning, but there is no way to track it across separate Kubeflow components. Having unified metadata tracking allows users to aggregate things like:

• Experiment runs
• Datasets
• Metrics
• Trained artifacts
• Hyperparameter configurations
• etc

Originally Kubeflow covered this through the Metadata project but it has since been archived. There were some additional discussions around this, found in issue kubeflow/kubeflow#4955.

It would be great to revisit this problem and see if we can propose a unified interface for metadata and artifact storage, possibly by using ML metadata.

Describe the solution you'd like:

One problem with the original Kubeflow metadata project is that it comes with its own storage backend using MySQL, which makes it heavy-weight. We do not need to re-implement the storage backend since MLMD already solves that problem. Instead, we can make MLMD an optional installation, and write to it directly. This is what KFP is currently doing, see this link for the code.

If we can define a unified data model and interface, it should be possible to build a light-weight library on top of ML metadata. It can be an optional import for training jobs and hyperparameter tuning jobs.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]