sqlmesh icon indicating copy to clipboard operation
sqlmesh copied to clipboard

Fix: Take batch_size into account when filling snapshots for INCREMENTAL_BY_UNIQUE_KEY models

Open erindru opened this issue 9 months ago • 1 comments

This PR makes the batch_index available when filling snapshots, so EvaluationStrategy's can take it into account when making decisions.

Prior to this, for kind's like INCREMENTAL_BY_UNIQUE_KEY, if:

  • You had a model that defines intervals (say @daily) and runs them in multiple batches (say, batch_size=1, which generates a Snapshot per interval)
  • When running the Snapshot's, SQLMesh would treat them all as "clear table -> insert data" rather than just the first one

After this change, for Snapshot's with no existing intervals at plan time, SQLMesh can check the index in the batch and just execute the "clear table" logic for the first snapshot and then the "merge into existing table" logic for subsequent snapshots

Fixes #2609

erindru avatar May 15 '24 22:05 erindru