sqlmesh
sqlmesh copied to clipboard
Feature request: Forced restatement of incremental models on merge to prod, e.g. using `stamp`
In my project, I use a lot of INCREMENTAL_BY_PARTITION kind models.
Note: This model type is not compatible with the forward_only FALSE option, despite the documentation saying this option is available for all PARTITION kinds.
To make corrections in these models that I want reflected for historical (in the incremental sense) records, I follow this workflow:
- Branch from main
- Make changes to model
- Push changes to GitHub, make PR
- Deploy using CI/CD bot
- Wait for PR to merge
- Pull main from origin
- Run local
sqlmesh plan --restate-model ...explicitly
Why is this workflow inelegant:
- Direct user interaction with the live production environment
- Have to remember to do a manual backfill at some point in the future after merge to prod (auto-restatement feature alleviates this, but as development matures this causes a lot of unnecessary scheduled restatements)
- Lack of in-code traceability in the PR that the model (will be) restated because of changes in said PR
Ideal workflow:
- Branch from main
- Make changes to model
- Update (e.g.)
stampvalue to signal that this is a breaking change and that it should be rebuilt - Push changes to GitHub, make PR
- Deploy using CI/CD bot