cmf icon indicating copy to clipboard operation
cmf copied to clipboard

Support for various metadata store backends [need feedback and review]

Open sergey-serebryakov opened this issue 3 years ago • 1 comments

In current implementation, the ML metadata (MLMD) metadata store was tightly coupled with CMF. Our experiments and standalone proof-of-concept implementations have demonstrated that it is possible to use other platforms, such as MLflow to store pipeline metadata (though, not as efficiently as with MLMD store, but in certain cases when a team or organization has already setup the ML metadata management service, they can start tracking their pipeline metadata using that existing service).

The idea is relatively straightforward - take MLMD related collection of functions and wrap them into a class (MlmdStore). Then functions that are used by Cmf become public interface for MlmdStore, and respectively, become API that Cmf requires from arbitrary metadata backend in the future.

This commit moves existing MLMD functionality into cmflib.metadata.mlmd_store module.

At this point, the implementation is merely a draft for the proposed idea. We are collecting feedback and suggestions.

sergey-serebryakov avatar Dec 04 '22 05:12 sergey-serebryakov

This PR should also address CMF running in the debug mode, where the backend can be a console , and we do a dry run and no hard changes like commits and branches .

annmary-roy avatar Jan 19 '23 15:01 annmary-roy