dbt-bigquery icon indicating copy to clipboard operation
dbt-bigquery copied to clipboard

[Bug] persist_docs not working for seeds

Open dpguthrie opened this issue 1 year ago • 2 comments

Is this a new bug in dbt-bigquery?

  • [X] I believe this is a new bug in dbt-bigquery
  • [X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

The descriptions for the seed and columns are not being persisted to BQ when using the appropriate config.

seeds:
  [<resource-path>](https://docs.getdbt.com/reference/resource-configs/resource-path):
    +persist_docs:
      relation: true
      columns: true

Expected Behavior

The descriptions are populated at the object and column level for the seed in my dbt project.

Steps To Reproduce

  1. Use the yaml below to persist_docs for seeds
seeds:
  [<resource-path>](https://docs.getdbt.com/reference/resource-configs/resource-path):
    +persist_docs:
      relation: true
      columns: true
  1. Run dbt seed
  2. View descriptions in BQ for that seed

This may have something to do with the size of the seed though. From @mikealfare "It might also have something to do with the fact that tables are sharded in BQ, and we only pull the column descriptions from the most recent shard (because they can change)."

Relevant log output

No response

Environment

- OS:
- Python:
- dbt-core:
- dbt-bigquery:

Additional Context

No response

dpguthrie avatar Apr 30 '24 22:04 dpguthrie

It looks like we persist top-level descriptions for seeds, but not column-level descriptions:

https://github.com/dbt-labs/dbt-bigquery/blob/b7bb0bc0818644617b8c10eb0723560ce98147a5/dbt/include/bigquery/macros/materializations/seed.sql#L20-L23

I'm not sure why that is - this implementation goes way back:

  • https://github.com/dbt-labs/dbt-core/pull/2601
Screenshot 2024-05-14 at 13 46 57

jtcohen6 avatar May 14 '24 11:05 jtcohen6

@jtcohen6 that's interesting, I was able to persist my column descriptions:

image

dpguthrie avatar May 28 '24 18:05 dpguthrie

@jtcohen6 @amychen1776 My customer is asking if we're able to prioritize this at all?

dpguthrie avatar Sep 26 '24 18:09 dpguthrie