soda-core
soda-core copied to clipboard
Dataset filters not compatible with Reference Checks
When trying to run a Soda Core reference check against a partitioned BigQuery table in combination with a filter, the filter is not passed into the SQL generated by SodaCore there by resulting an execution error.
For example,
filter my_project.my_dataset.source_table [daily]:
where: dt = '2023-02-01'
checks for my_project.my_dataset.my_table [daily]:
- values in (id) must exist in my_project.my_dataset.target_table (id)
Assuming that the table source_table
is partitioned on dt
, then this will result in the following error.
[2023-03-02 11:37:01,376] {log.py:101} ERROR - [11:37:01] Query execution error in my_project.my_dataset.source_table[daily].reference[id]: 400 Cannot query over table `my_project.my_dataset.source_table` without a filter over column(s) 'dt' that can be used for partition elimination