dbt-bigquery
dbt-bigquery copied to clipboard
[ADAP-369] [Feature] Include all bytes processed in the adapter response
Is this a regression in a recent version of dbt-bigquery?
- [X] I believe this is a regression in dbt-bigquery functionality
- [X] I have searched the existing issues, and I could not find an existing issue for this regression
Current Behavior
Hey folks this is a bit of a convoluted "regression" so I'm happy to mark this as something else but following the merge of #77 I noticed that the bytes_processed
returned in the adapter response for models with copy_partitions: true
is now 0. This kind of makes sense, bigquery copies are free so 0 bytes is kind of correct but its ultimately unexpected because the construction of the temporary table is ultimately not free.
This is indicative of a larger problem with the adapter response in that it only returns the bytes processed for the last step, it does not consider every part of running the model. You can see this when copy_partitions: false
, dbt is under reporting the cost by up to half of running incremental models because we big-query will charge you for both the creation of a temporary table and then again to read the temporary table for the merge query into the destination.
Expected/Previous Behavior
given a similar config to:
model_name:
+materialized: incremental
+partition_by:
copy_partitions:
field: timestamp
data_type: timestamp
granularity: day
On a dbt run I'd expect to see the bytes processed be equal to the amount of bytes that would be processed by running the target/compiled/.../model_name.sql
file in the bigquery console, instead I see:
19:20:57 1 of 1 OK created sql incremental model dataset.model_name................ [None (0 processed) in 166.72s]
Steps To Reproduce
see above
Relevant log output
No response
Environment
- OS:Ubuntu 18.04.6 LTS (Bionic Beaver)
- Python: 3.11.2
- dbt-core (working version): N/A
- dbt-bigquery (working version): N/A
- dbt-core (regression version): 1.5.0-b3
- dbt-bigquery (regression version): 1.5.0b2
Additional Context
No response