The database file was created with an newer version of DuckDB.

[ibayer]

Thanks for developing both dbt-duckdb and target-duckdb!

Being new to the whole meltano, dbt, duckdb stack I tried to modify the official meltanto tutorial by replacing postgres with duckdb

cat meltano.yml 
version: 1
default_environment: dev
project_id: b1cb5b90-6956-4199-b7ed-e10092dfb5da
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-github
    variant: singer-io
    pip_url: tap-github
    config:
      repository: palaimon/ipyannotator palaimon/fastfm2
      start_date: ' 2016-11-28'
    select:
    - commits.url
    - commits.sha
    - commits.*
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
  - name: target-duckdb
    variant: jwills
    pip_url: target-duckdb~=0.4
    config:
      add_metadata_columns: true
      filepath: dump.duckdb
      default_target_schema: tap_github
  transformers:
  - name: dbt-duckdb
    variant: jwills
    pip_url: dbt-core~=1.2.0 dbt-duckdb~=1.2.0
    config:
      path: <my local path>/dump.duckdb

and run into the following error message.

dbt.exceptions.RuntimeException: Runtime Error
  IO Error: Trying to read a database file with version number 39, but we can only read version 38.
  The database file was created with an newer version of DuckDB.
  
  The storage of DuckDB is not yet stable; newer versions of DuckDB cannot read old database files and vice versa.
  The storage will be stabilized when version 1.0 releases.
  
  For now, we recommend that you load the database file in a supported version of DuckDB, and use the EXPORT DATABASE command followed by IMPORT DATABASE on the current version of DuckDB.

The error Trying to read a database file with version number 39, but we can only read version 38. seems pretty clear but I wasn't able to find a way to pin the dbt-duckdb and target-duckdb version in such a way that they are compatible.

A pointer on how to make dbt-duckdb and target-duckdb work together in meltano would be great.

[jwills]

Ah, that's a good (and important!) question that I don't know the answer to; let me ask about it in the Meltano Slack!

[jwills]

There's also an argument to be made that I should ditch Meltano and create my own DuckDB-centric MDS in a box project where I design all of the bits to work together 🤔 /cc @matsonj

[jwills]

Jacob's recommendation is to use target-parquet instead of target-duckdb to get around the issue, and I am thinking that is the best short-term strategy here until I actually build that MDS in a box project, which tbh I am warming up to as I sit here thinking about it 🤔

[ibayer]

Jacob's recommendation is to use target-parquet instead of target-duckdb to get around the issue,

Thanks for sharing, I'll try it out. Such workarounds make we wonder if the added complexity of Meltano is worth the ~~confidence~~ convenience benefits. Let's see how well the integration of airflow in Meltano works. :)

I actually build that MDS in a box project, which tbh I am warming up to as I sit here thinking about it thinking

I would be super excited to learn about it.

Having just read https://duckdb.org/2022/10/12/modern-data-stack-in-a-box.html I'm still in a very early stage of my DuckDB-centric MDS journey :)

[matsonj]

@ibayer imo the advantage of meltano is the ecosystem of connectors - ironically the csv extractor is not required to use with dbt-duckdb. Think about the MDS in a box project as more like a frame in which you can swap pieces in and out as required. For me, I will be swapping out the data viz bits as well as some new extractors. Happy to chat here or in twitter DMs or what have you if you have questions!!

[potter-potter]

I noticed that target-duckdb installed duckdb 0.7.1 So I just pinned the dbt-duckdb setup.py at 0.7.1 (it was at 0.5.1 which caused the problem) by downloading and pip installing from local. And then didn't have any problem using dbt-duckdb. I didn't do a thorough test, but the MDS in a box that I was making worked fine.

So maybe if the duckdb in the setup.py for both the target and dbt are the same, it will work for others?

[ReneTC]

any updates on this one? I have the same problem.