dbt-bigquery-monitoring icon indicating copy to clipboard operation
dbt-bigquery-monitoring copied to clipboard

Limitation/"Bug": INFORMATION_SCHEMA.PARTITIONS requests are limited to 1000 tables

Open Kayrnt opened this issue 1 year ago • 1 comments

See https://cloud.google.com/bigquery/docs/information-schema-partitions It leads to failing executions of models on setup with more than 1000 tables

Kayrnt avatar Oct 25 '24 14:10 Kayrnt

As there's no immediate workaround (maybe through a dedicated materialization like for project one). In the meantime, if you run into the issue, you can disable the related model by adding in your configuration (dbt_project.yml) :

...
models:
  dbt_bigquery_monitoring:
    monitoring:
      storage:
        datamart:
          table:
            partitions_monitoring:
              +enabled: false
        intermediate:
          stg_partitions_monitoring:
            +enabled: false

Kayrnt avatar Feb 03 '25 15:02 Kayrnt