dbt-bigquery
dbt-bigquery copied to clipboard
[Feature] `state:modified` should pick up changes to policy tags when `persist_docs: true`
Is this a new bug in dbt-bigquery?
- [X] I believe this is a new bug in dbt-bigquery
- [X] I have searched the existing issues, and I could not find an existing issue for this bug
Current Behavior
When editing the yml of a model, adding a BigQuery policy tag does not get recognised as a state change.
Expected Behavior
I would expect that this change it picked up by dbt when I run state:modified. Also because this is not flagged as an exception in the documentation.
Steps To Reproduce
We start with this, as part of the model.yml
...
- name: cost
description: Total cost of the campaign.
...
Then make a change to
...
- name: cost
description: Total cost of the campaign.
policy_tags:
- 'projects/sample_project/locations/europe-west1/taxonomies/903457/policyTags/3470345'
...
When you recompile and run dbt run -s state:modified --defer
, dbt won't see any changes.
But when we change the description, this will be picked up:
...
- name: cost
description: Total cost of the campaign, protected by random tag.
policy_tags:
- 'projects/sample_project/locations/europe-west1/taxonomies/903457/policyTags/3470345'
...
Relevant log output
08:55:08 Found 121 models, 2 operations, 1 seed, 86 tests, 20 sources, 0 exposures, 0 metrics, 1257 macros, 0 groups, 0 semantic models
08:55:08 The selection criterion 'state:modified+' does not match any nodes
08:55:08
08:55:08 Nothing to do. Try checking your model configs and model specification args
Environment
Tested on:
- OS: MacOS
- Python: 3.11.1
- dbt-core: 1.7.10
- dbt-bigquery: 1.7.7
and
- OS:Debian
- Python: 3.12
- dbt-core: 1.7.10
- dbt-bigquery: 1.7.7
Additional Context
No response
After a discussion internally, I would view this as a feature request rather than a bug. This is because of how we have implemented state:modified.
- We will check for state:modified if persist_docs is also applied.
- However, today, we only check for descriptions changed rather than inclusive of all the column level configurations.
- On BQ, this should include policy_tags.
- What makes this mesy is this is also defined on dbt-core and not on dbt-adapters so we will need to see who owns this implementation. @dbeatty10 @graciegoheen I'm very interested in your thoughts here :)
4. What makes this messy is this is also defined on dbt-core and not on dbt-adapters so we will need to see who owns this implementation.
Yeah, that is tricky @amychen1776!
One idea:
Adapters can specify the incremental strategies that they support (example). What if they could similarly define the properties that contribute towards calculation of state:modified
for a dbt model? And then if dbt-core took that into account in a place like here?