ML project discovery using Amundsen

[dwarszawski]

Summary

Our proposal is to extend the capabilities of Amundsen by introducing new data resources covering the ML project lifecycle.

Expected Behavior or Use Case

Ability to discover ml projects, ml deployments, ml artifacts or model owners

Possible Implementation

Apache Atlas introduces dedicated entities for ML governance in version 2.1.0. That can be adapted for the purpose of ml discovery including changes in the frontend, search, metadata components. Please refer to this docs which describes the structure of Atlas entities

Context

The rising number of ML projects force organizations to address challenges related to model governance and model discovery. According to latest market surveys one can expect significant acceleration in AI adoption this and next year.

[feng-tao]

thank @dwarszawski , this is very exciting? how about creating a rfc in https://github.com/amundsen-io/rfcs and document the detail approach? E.g I am curious to know any plan / UI mock for ML feature as well as what is the plan for ingestion?

[dwarszawski]

initial draft of RFC available here

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.