sqlmesh
sqlmesh copied to clipboard
Fix: Take batch_size into account when filling snapshots for INCREMENTAL_BY_UNIQUE_KEY models
This PR makes the batch_index
available when filling snapshots, so EvaluationStrategy
's can take it into account when making decisions.
Prior to this, for kind
's like INCREMENTAL_BY_UNIQUE_KEY, if:
- You had a model that defines intervals (say
@daily
) and runs them in multiple batches (say,batch_size=1
, which generates aSnapshot
per interval) - When running the
Snapshot
's, SQLMesh would treat them all as "clear table -> insert data" rather than just the first one
After this change, for Snapshot
's with no existing intervals at plan time, SQLMesh can check the index in the batch and just execute the "clear table" logic for the first snapshot and then the "merge into existing table" logic for subsequent snapshots
Fixes #2609