OpenMetadata icon indicating copy to clipboard operation
OpenMetadata copied to clipboard

Published 20 hours ago • verified publisher icon open-metadata

Ingest more metadata from DBT

Open geoHeil opened this issue 2 years ago • 3 comments

DBT allows to add much more metadata to its assets than only column names and descriptions and lineage.

In particular it allows to specify data owners, additional key-value metadata pairs and so on:

  • https://docs.getdbt.com/reference/resource-configs/meta

In fact with the recent upgrade around exposures:

  • https://github.com/gouline/dbt-metabase
  • https://docs.getdbt.com/docs/building-a-dbt-project/exposures

several BI visualization tools started to integrate around these features (metabase, mode, looker, superset)

  • https://github.com/slidoapp/dbt-superset-lineage
  • https://mode.com/get-dbt/
  • https://www.getdbt.com/blog/how-to-integrate-dbt-and-looker-with-user-attributes/

It would be great if open-metadata could ingest these additional attributes as well.

geoHeil avatar Aug 03 '22 14:08 geoHeil

https://openmetadata.slack.com/archives/C02B6955S4S/p1659539501092759?thread_ts=1659537963.701169&cid=C02B6955S4S DBTs schema.yaml` which is defining documentation around column metadata does not seem to be imported as well. I think this would be most important to get these semantics imported.

geoHeil avatar Aug 03 '22 15:08 geoHeil

One more great missing integration is dbt test. Ideally, one does not need to define additional tests in OpneMetadata - rather the ones from DBT can be integrated.

geoHeil avatar Aug 04 '22 03:08 geoHeil

DBT tests will be added in https://github.com/open-metadata/OpenMetadata/issues/6135

pmbrull avatar Aug 04 '22 04:08 pmbrull

Created https://github.com/open-metadata/OpenMetadata/issues/6801 to follow up on column descriptions as well as owner.

we will use this issue to discuss on custom properties from the ingestion

pmbrull avatar Aug 18 '22 13:08 pmbrull

Hi @geoHeil We are adding below features in our DBT integration as a part of 0.12.1 release

  • Ingesting DBT Tests and test results information
  • Assigning owner to a model/table in OpenMetadata from DBT. Docs Link
  • Adding table level and column level tags from DBT to OpenMetadata
  • Adding a control on whether table and column descriptions from DBT should overwrite the already present descriptions

We will be taking into consideration the other features listed in this issue for our 0.13 release. Thank you for your contribution and let us know if you have any doubts/queries.

OnkarVO7 avatar Sep 26 '22 15:09 OnkarVO7

@OnkarVO7 what is remaining to close this ticket?

harshach avatar Oct 12 '22 23:10 harshach

@harshach let me make an aggregate list of the things remaining and will update here

OnkarVO7 avatar Oct 13 '22 05:10 OnkarVO7

ingesting exposure and metrics would be amazing. thanks

mingfang avatar Oct 19 '22 23:10 mingfang

@OnkarVO7 please provide an update here

harshach avatar Dec 10 '22 19:12 harshach

hi @OnkarVO7 if you could please confirm, owner and tags from meta fields are already being ingested. The only missing point from this issue is around exposures. Is that correct?

pmbrull avatar Jan 04 '23 09:01 pmbrull

@pmbrull yes owner and tags have been implemented

OnkarVO7 avatar Jan 04 '23 09:01 OnkarVO7

Thanks, then I'll be closing this issue and listing down exposures in the EPIC https://github.com/open-metadata/OpenMetadata/issues/8486

pmbrull avatar Jan 04 '23 10:01 pmbrull