Add DBT integration for better tests, documentation, and version-control of data transformations

[milicictanja]

Is your feature request related to a problem? Please describe. My team is currently using DBT for batch data processing, versioning and documenting of our data transformations. The tool is doing a great job in batch processing, however it can't work with streams of data. KsqlDB is a great tool that can help us process real-time data, however we would need to migrate all of our models to ksqlDB format. It would be great if ksqlDB would have dbt adapter that would translate dbt models into ksqlDB streams. Another similar tool Materialize supports that, however it doesn't have the option for self-hosting.

Describe the solution you'd like Implement a DBT adapter similar to dbt-materialize

[lcorneliussen]

+1

Would love a good solution of using source control to maintain (and test + document) ksql. Dbt looks like a very natural fit.

[daigotanaka]

Hi all,

I started experimenting dbt-ksql here: https://github.com/anelendata/dbt-ksql I also attached an example project: https://github.com/anelendata/dbt-ksql/tree/main/example

I just coded this in a day, so it's still experimental, but my productivity has been boosted for sure.

Feedback and collaborations are welcome :)

[jordanhamel]

Any updates or thoughts on this open issue? @agavra thanks