oso Clickhouse sqlmesh pipeline

What is it?

dbt pipelines do not span warehouses.

Since we are already looking to do more stuff in Clickhouse, we can use this as a testing ground for sqlmesh. This is very P2 for the time being.

I filed a separate issue to move data in between the 2 pipelines: https://github.com/opensource-observer/oso/issues/1164

Apr 02 '24 04:04 ryscheng

Deprioritizing this

Apr 03 '24 22:04 ryscheng

Probably doesn't cover 100% of the models as Clickhouse needs a somewhat different approach than BigQuery but SQLGlot might be useful for the base models (can run at the Dagster level).

Apr 04 '24 17:04 davidgasquez

column lineage - seeing schema changes
test-deploy / playground in duckdb
generated models per network

Jul 17 '24 22:07 ryscheng

Going to backlog this for now, sqlmesh is working on Clickhouse integrations, but its not ready yet. There's only BigQuery, duckdb, and postgres

Aug 21 '24 22:08 ryscheng

Updated scope: @ravenac95 got this working off a sqlmesh branch. Remaining work is to have it orchestrated by Dagster

Sep 02 '24 16:09 ryscheng