mltrace
mltrace copied to clipboard
Add support or follow the OpenLineage Spec
For the components steps, MlTrace can follow/implement the OpenLineage Spec https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md.
- Offer an endpoint to allow Metadata Engines (DataHub or Marquez or Amunsden) to collect Traces.
- Record versionned Data provenance from LakesFS, TeminusDB, Dolt, DBTSnapShots
Ref:
- https://github.com/OpenLineage/OpenLineage
- https://github.com/treeverse/lakeFS
- https://github.com/terminusdb/terminusdb
- https://github.com/dolthub/dolt
- https://docs.getdbt.com/docs/building-a-dbt-project/snapshots
Putting it on the backlog!