soda-core icon indicating copy to clipboard operation
soda-core copied to clipboard

Dataset filters not compatible with Reference Checks

Open nathadfield opened this issue 1 year ago • 1 comments

When trying to run a Soda Core reference check against a partitioned BigQuery table in combination with a filter, the filter is not passed into the SQL generated by SodaCore there by resulting an execution error.

For example,

filter my_project.my_dataset.source_table [daily]:
  where: dt = '2023-02-01'

checks for my_project.my_dataset.my_table [daily]:
  - values in (id) must exist in my_project.my_dataset.target_table (id)

Assuming that the table source_table is partitioned on dt, then this will result in the following error.

[2023-03-02 11:37:01,376] {log.py:101} ERROR - [11:37:01] Query execution error in my_project.my_dataset.source_table[daily].reference[id]: 400 Cannot query over table `my_project.my_dataset.source_table` without a filter over column(s) 'dt' that can be used for partition elimination

nathadfield avatar Mar 02 '23 13:03 nathadfield