dbt-duckdb icon indicating copy to clipboard operation
dbt-duckdb copied to clipboard

Cannot reference attached DuckDB databases in sources.yml

Open Phil-T1 opened this issue 1 year ago • 4 comments

Hi,

I appreciate that this is probably something that I am missing, but I cannot see how it is possible to reference an attached DuckDB db in sources.yml?

profiles.yml:

attach:
  -path: "./attached_database.db"

I can successfully select from the attached database using the below without Jinja (double quotes must be included here): SELECT * FROM "attached_database"."attached_table"

However, if I use the following in sources.yml:

  - name: attached_source
    database: attached_database
    tables:
      - name: attached_table

Then SELECT * FROM {{ source('attached_source', 'attached_table') }} Compiles to and fails: SELECT * FROM "attached_source"."attached_database"."attached_table"

If I leave the database part out in sources.yml and use schema:

  - name: attached_source
    schema: attached_database
    tables:
      - name: attached_table

Then SELECT * FROM {{ source('attached_source', 'attached_table') }} Of course it compiles to (and fails): SELECT * FROM "main"."attached_database"."attached_table"

How do I correctly set sources.yml or correctly reference sources.yml to have a select compile to "attached_database"."attached_table"?

Thanks!

Phil.

Phil-T1 avatar Oct 04 '23 19:10 Phil-T1

You need to use both the "database" and the "schema" properties-- I'm assuming what you want is:

- name: attached_source
   database: attached_database
   schema: main

jwills avatar Oct 04 '23 20:10 jwills

(If that doesn't work-- i.e., it does not then compile to "attached_database.main.attached_table", it's likely a bug)

jwills avatar Oct 04 '23 20:10 jwills

Thanks for the speedy response again Josh, much appreciated :)

I couldn't get it to work with your solution, but once you told me I should be aiming for compilation to: "attached_database"."main"."attached_table" then I managed to get it to work using:

sources:

  - name: attached_database
    schema: main
    database: attached_database
    tables:
      - name: attached_table

I had to try quite a few permutations to get this to succeed, is it worth updating the docs to give an example?

Thanks again,

Phil.

Phil-T1 avatar Oct 05 '23 06:10 Phil-T1

Ah gotcha— yeah, the list of tables in the source is a key part of the config. An example for the README would be most welcome!

jwills avatar Oct 05 '23 16:10 jwills