sqlmesh
sqlmesh copied to clipboard
TobikoData
Efficient data transformation and modeling framework that is backwards compatible with dbt.
This PR makes the `batch_index` available when filling snapshots, so `EvaluationStrategy`'s can take it into account when making decisions. Prior to this, for `kind`'s like INCREMENTAL_BY_UNIQUE_KEY, if: - You had...
**Tasks** - [x] Model defaults code, tests - [x] Plan selector code, tests - [x] Plan time code, tests - [x] Run time code, tests - [x] Docs - [...
On Trino/Iceberg (and potentially other engines; unconfirmed), the following model doesnt backfill correctly: ``` MODEL ( name datalake_iceberg.stg_premium_day4, kind INCREMENTAL_BY_UNIQUE_KEY ( unique_key id, batch_size 1 ), start '2020-01-01', cron '@daily'...
when running `sqlmesh plan` I got this error: ``` AttributeError: 'str' object has no attribute 'is_type' Error: Failed to resolve macro for expression. Error trying to eval macro. ``` if...
The scenario goes as follows: 1. table which is defined as an external source was ingested and serves as a base table 2. that table has some meta fields that...
based on the following dbt source yaml setup, sqlmesh will raise error with `Could not render or parse jinja at '/Users/leizhou/Dev/dbt_park/models/raw-sparcs/_sources.yml'. Encountered unknown tag 'test'.` ``` --- version: 2 sources:...
- [x] Add option for allowing empty / unspecified columns in unit tests (related Slack thread [here](https://tobiko-data.slack.com/archives/C044BRE5W4S/p1698620706683929)) The use-case for this is that one can have wide tables but they...
Currently, user-defined macros need to be defined as python. An alternative SQL-based syntax would be more consistent with `MODEL` and `AUDIT`. That is, having a `MACRO` function to define user-defines...
We currently create temporary tables at plan time preemptively, even though they may not necessarily be needed. This behavior was introduced before we started to create tables on demand (i.e.,...
Currently, sqlmesh supports synchronizing with externally scheduled dependencies in the Airflow scheduler via "[external signals](https://sqlmesh.readthedocs.io/en/latest/integrations/airflow/#external-signals)". That is, one can define an "external signal" on a sqlmesh model which allows Airflow...
Metadata
Owner
Metadata
Efficient data transformation and modeling framework that is backwards compatible with dbt.