dbt-clickhouse icon indicating copy to clipboard operation
dbt-clickhouse copied to clipboard

First version of distributed materialization

Open Volodin-DD opened this issue 2 years ago • 2 comments

NOTE: lightly tested (though successfully). Need help from community.

Made two materializations: distributed_table for distributed version of table models and distributed_incremental for incremental models.

The main idea is to make local tables on cluster with engine and parameters from config in model's sql file within a distributed table "looking" on these local tables. So when referencing model with distributed materialization dbt will make insert into distributed table and select from distributed table. Sharding key is also to be set on model's config.

Main flow described in comments distributed_incremental.sql

Volodin-DD avatar Aug 27 '21 10:08 Volodin-DD

There were some issues with schemas. Fixed it

Volodin-DD avatar Nov 17 '21 17:11 Volodin-DD

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

:white_check_mark: Volodin-DD
:x: Dimitry Volodin


Dimitry Volodin seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

CLAassistant avatar Jun 05 '22 10:06 CLAassistant

Closed in favor of #172 and #186

genzgd avatar Nov 30 '23 14:11 genzgd