sqlmesh icon indicating copy to clipboard operation
sqlmesh copied to clipboard

Feat: Support SQLMesh project generation from dlt pipeline

Open Themiscodes opened this issue 4 months ago • 1 comments

WIP: This update introduces the ability to generate a SQLMesh project from a dlt pipeline. It creates the project directory scaffolding, inspects the schema and automatically generates incremental models from the tables and sets up the config.yaml connection configurations using the pipeline's credentials.

Importing a DLT project:

To import a dlt project into SQLMesh, ensure the dlt pipeline has been run or restored locally. Then, in the pipeline directory, use the init command specifying its name:

sqlmesh init -t dlt --dlt-pipeline "pipeline_name" dialect

The resulting SQLMesh project can be executed as usual with sqlmesh plan. The incremental models are designed to process data using the unique _dlt_load_id key present in each dlt table, but the model definitions can be modified as needed.

Themiscodes avatar Oct 03 '24 18:10 Themiscodes