stitchfix
Metadata emission
Is your feature request related to a problem? Please describe. Hamilton encodes a lot of metadata that lives in code. It also creates some at execution time. There are projects such as https://datahubproject.io/, https://openlineage.io/ that capture this metadata across a wide array of tooling to create a central view in a heterogenous environment. Hamilton should be able to emit metadata to them.
Describe the solution you'd like A user should be able to specify whether their Hamilton DAG should emit metadata. This should play nicely with graph adapters, e.g. spark, ray, dask.
UX questions:
- Should this be something in the graph adapter universe? E.g. a mixin?
- Or should this be on the driver side, so you change drivers for functionality, but change graph adapters for scale...
Adding a custom source for Datahub:
- https://datahubproject.io/docs/metadata-ingestion/developing/
- https://datahubproject.io/docs/metadata-ingestion/adding-source/
FYI - @gravesee - seems like I created this a while back for emission of metadata/usage.
Would love any help in providing a motivating use case!
We are moving repositories! Please see the new version of this issue at https://github.com/DAGWorks-Inc/hamilton/issues/30. Also, please give us a star/update any of your internal links.
