Feature request: Sync dbt model tags with Databricks table/view tags

[AliAl-Gburi]

trafficstars

Description

Dbt allow you to tag your models, and you can also tag your tables and views in Databricks. There should be a way to apply dbt tags to the tables and views in Databricks.

Alternative

I have considered implementing a custom solution using:

1. Python to read the tags from the dbt models
2. Databricks-sql-cli to read the tags from databricks
3. Then databricks-sql-cli again to update the tags in databricks such that they are aligned with the dbt tags

Benefits

This is useful for filtering the tables in databricks.

I would like to contribute in creating a seamless syncing process.

[benc-db]

@AliAl-Gburi this is something we are thinking about as well. I think where things are a little complicated is that some dbt tags are strictly for dbt operations (e.g. I tag some tables with 'daily' for scheduling daily runs), but maybe it doesn't matter if those get synced to Databricks.

What are your thoughts on tags in Databricks that are not found in the dbt project? Specifically, how can a dbt project indicate that it wants to remove a tag? We have a similar issue with materialized views, where a tblproperty gets set by Databricks and we have to figure out the meaning of the absence of that tblproperty in the dbt project.

[AliAl-Gburi]

Hey @benc-db, thanks for the answer :D. For those who want dbt tags to remain in dbt and not overwrite whatever tags they've set in Databricks, the option to sync tags can be turned off.

I suppose "syncing" was not the correct term to use here. The idea is to have dbt tags be your single source of truth and then Databricks tables and views would match the tags defined in dbt.

[benc-db]

Hi @AliAl-Gburi. I've got a PR for this now, for our 2.0.0 release (aligning with the dbt-core 1.8.0 release).

[AliAl-Gburi]

Thats great to hear, thanks a lot :D

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.