mltrace icon indicating copy to clipboard operation
mltrace copied to clipboard

Add support or follow the OpenLineage Spec

Open MChorfa opened this issue 4 years ago • 1 comments

For the components steps, MlTrace can follow/implement the OpenLineage Spec https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md.

  • Offer an endpoint to allow Metadata Engines (DataHub or Marquez or Amunsden) to collect Traces.
  • Record versionned Data provenance from LakesFS, TeminusDB, Dolt, DBTSnapShots

Ref:

  • https://github.com/OpenLineage/OpenLineage
  • https://github.com/treeverse/lakeFS
  • https://github.com/terminusdb/terminusdb
  • https://github.com/dolthub/dolt
  • https://docs.getdbt.com/docs/building-a-dbt-project/snapshots

MChorfa avatar Oct 07 '21 13:10 MChorfa

Putting it on the backlog!

shreyashankar avatar Oct 24 '21 07:10 shreyashankar