[ADAP-555] [Spike] Capture the number of partitions pruned when running an incremental model

[dbeatty10]

Is this your first time submitting a feature request?

• [X] I have read the expectations for open source contributors
• [X] I have searched the existing issues, and I could not find an existing issue for this feature
• [X] I am requesting a straightforward extension of existing dbt-bigquery functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Research methods to capture the total number of partitions scanned when running an incremental model.

A potential approach:

1. Determine how many partitions are within the table for an incremental model
   • using TABLE_STORAGE view similar to https://github.com/dbt-labs/dbt-bigquery/pull/364
2. Determine how many partitions are scanned by the queries for an incremental model
   • using query_job.job_id within Client.get_job then statistics.query.totalPartitionsProcessed JSON path (example)
3. Determine how many partitions are pruned by a query by subtracting 2) from 1)

Describe alternatives you've considered

I'm not sure if the BigQuery web interface will show partition counts or not. If so, looking up jobs via the job_id in the dbt logs could be a (very manual) alternative.

Who will this benefit?

This would help reproduce and troubleshoot issues like https://github.com/dbt-labs/dbt-bigquery/issues/708 and https://github.com/dbt-labs/dbt-bigquery/issues/717 (and test their solutions).

Are you interested in contributing this feature?

No response

Anything else?

No response

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions[bot]]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.