dbt-bigquery icon indicating copy to clipboard operation
dbt-bigquery copied to clipboard

[Feature] `state:modified` should pick up changes to policy tags when `persist_docs: true`

Open kverburg opened this issue 10 months ago • 2 comments

Is this a new bug in dbt-bigquery?

  • [X] I believe this is a new bug in dbt-bigquery
  • [X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When editing the yml of a model, adding a BigQuery policy tag does not get recognised as a state change.

Expected Behavior

I would expect that this change it picked up by dbt when I run state:modified. Also because this is not flagged as an exception in the documentation.

Steps To Reproduce

We start with this, as part of the model.yml

...
      - name: cost
        description: Total cost of the campaign.
...

Then make a change to

...
      - name: cost
        description: Total cost of the campaign.
        policy_tags:
          - 'projects/sample_project/locations/europe-west1/taxonomies/903457/policyTags/3470345'
...

When you recompile and run dbt run -s state:modified --defer , dbt won't see any changes.

But when we change the description, this will be picked up:

...
      - name: cost
        description: Total cost of the campaign, protected by random tag.
        policy_tags:
          - 'projects/sample_project/locations/europe-west1/taxonomies/903457/policyTags/3470345'
...

Relevant log output

08:55:08  Found 121 models, 2 operations, 1 seed, 86 tests, 20 sources, 0 exposures, 0 metrics, 1257 macros, 0 groups, 0 semantic models
08:55:08  The selection criterion 'state:modified+' does not match any nodes
08:55:08  
08:55:08  Nothing to do. Try checking your model configs and model specification args

Environment

Tested on:
- OS: MacOS
- Python: 3.11.1
- dbt-core: 1.7.10
- dbt-bigquery: 1.7.7
and
- OS:Debian
- Python: 3.12
- dbt-core: 1.7.10
- dbt-bigquery: 1.7.7

Additional Context

No response

kverburg avatar Apr 24 '24 10:04 kverburg

After a discussion internally, I would view this as a feature request rather than a bug. This is because of how we have implemented state:modified.

  1. We will check for state:modified if persist_docs is also applied.
  2. However, today, we only check for descriptions changed rather than inclusive of all the column level configurations.
  3. On BQ, this should include policy_tags.
  4. What makes this mesy is this is also defined on dbt-core and not on dbt-adapters so we will need to see who owns this implementation. @dbeatty10 @graciegoheen I'm very interested in your thoughts here :)

amychen1776 avatar Aug 02 '24 12:08 amychen1776

4. What makes this messy is this is also defined on dbt-core and not on dbt-adapters so we will need to see who owns this implementation.

Yeah, that is tricky @amychen1776!

One idea:

Adapters can specify the incremental strategies that they support (example). What if they could similarly define the properties that contribute towards calculation of state:modified for a dbt model? And then if dbt-core took that into account in a place like here?

dbeatty10 avatar Aug 04 '24 22:08 dbeatty10