dbt-bigquery icon indicating copy to clipboard operation
dbt-bigquery copied to clipboard

[ADAP-369] [Feature] Include all bytes processed in the adapter response

Open abuckenheimer opened this issue 1 year ago • 6 comments

Is this a regression in a recent version of dbt-bigquery?

  • [X] I believe this is a regression in dbt-bigquery functionality
  • [X] I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

Hey folks this is a bit of a convoluted "regression" so I'm happy to mark this as something else but following the merge of #77 I noticed that the bytes_processed returned in the adapter response for models with copy_partitions: true is now 0. This kind of makes sense, bigquery copies are free so 0 bytes is kind of correct but its ultimately unexpected because the construction of the temporary table is ultimately not free.

This is indicative of a larger problem with the adapter response in that it only returns the bytes processed for the last step, it does not consider every part of running the model. You can see this when copy_partitions: false, dbt is under reporting the cost by up to half of running incremental models because we big-query will charge you for both the creation of a temporary table and then again to read the temporary table for the merge query into the destination.

Expected/Previous Behavior

given a similar config to:

model_name:
    +materialized: incremental
    +partition_by:
              copy_partitions: 
              field: timestamp
              data_type: timestamp
              granularity: day

On a dbt run I'd expect to see the bytes processed be equal to the amount of bytes that would be processed by running the target/compiled/.../model_name.sql file in the bigquery console, instead I see:

19:20:57  1 of 1 OK created sql incremental model dataset.model_name................ [None (0 processed) in 166.72s]

Steps To Reproduce

see above

Relevant log output

No response

Environment

- OS:Ubuntu 18.04.6 LTS (Bionic Beaver)
- Python: 3.11.2
- dbt-core (working version): N/A
- dbt-bigquery (working version): N/A 
- dbt-core (regression version): 1.5.0-b3
- dbt-bigquery (regression version): 1.5.0b2

Additional Context

No response

abuckenheimer avatar Mar 14 '23 19:03 abuckenheimer