dbt-duckdb

dbt-duckdb copied to clipboard

As md environment differs slightly (and hopefully less in the future ;)) from duckDB, I was thinking of creating a dedicated environment class that would be loaded when users specify a path starting with md: I'm thinking of the following :

• Check for md token is present
• Disable transaction by default as not supported.
• Keep a list of supported extensions by md and warn the users if this goes outside these.

WDYT? Happy to take a stab at it 🙂

This would be great @mehd-io -- please send an impl my way!

Couple of things for you to keep in mind:

• You also likely want to disable any plugins, fsspec-based filesysems, and Python jobs-- i.e., in your environment, trying to call load_source or submit_python_job should raise an exception.
• Even better would be a way to indicate in the table materialization that Python models aren't supported in MD so that the failure would happen at compile time instead of runtime.
• It would also be nice to have first-class support for some MD constructs in non-MD environments (e.g. local or Buena Vista)-- i.e., being able to cleanly attach an MD share, or upload a local DuckDB file to MD at the end of a dbt-duckdb run.

Oh and please add an integration test profile for your MD environment so we can run it as part of CI