sqlmesh
sqlmesh copied to clipboard
Feat: Support SQLMesh project generation from dlt pipeline
WIP: This update introduces the ability to generate a SQLMesh project from a dlt pipeline. It creates the project directory scaffolding, inspects the schema and automatically generates incremental models from the tables and sets up the config.yaml
connection configurations using the pipeline's credentials.
Importing a DLT project:
To import a dlt project into SQLMesh, ensure the dlt pipeline has been run or restored locally. Then, in the pipeline directory, use the init
command specifying its name:
sqlmesh init -t dlt --dlt-pipeline "pipeline_name" dialect
The resulting SQLMesh project can be executed as usual with sqlmesh plan
. The incremental models are designed to process data using the unique _dlt_load_id
key present in each dlt table, but the model definitions can be modified as needed.