airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

Update DuckDB connector for destinations V2

Open guenp opened this issue 1 year ago • 2 comments

What

Enable Destinations V2 support for the DuckDB connector

How

  • Migrated a few files (constants.py, exceptions.py, name_normalizers.py and types.py) from the pyairbyte repo to airbyte-cdk/python/airbyte_cdk/sql
  • Added new _get_sql_column_definitions method to DestinationDuckDb based on pyairbyte's SqlProcessorBase._get_sql_column_definitions
  • Add SQLAlchemy and rich dependencies
  • Moved all tests into tests/ folder to make it easier to set up VS Code

Review guide

  1. airbyte-cdk/python/airbyte_cdk/sql/constants.py: copied from PyAirbyte
  2. airbyte-cdk/python/airbyte_cdk/sql/exceptions.py: copied from PyAirbyte
  3. airbyte-cdk/python/airbyte_cdk/sql/name_normalizers.py: copied from PyAirbyte
  4. airbyte-cdk/python/airbyte_cdk/sql/types.py: copied from PyAirbyte
  5. airbyte-integrations/connectors/destination-duckdb/destination_duckdb/destination.py: added _get_sql_column_definitions and various other changes

User Impact

Airbyte will no longer create tables with column names _airbyte_ab_id, _airbyte_emitted_at, _airbyte_data, but instead _airbyte_raw_id, _airbyte_extracted_at, _airbyte_meta plus any columns defined in the JSON schema.

Can this PR be safely reverted and rolled back?

  • [x] YES 💚
  • [ ] NO ❌

guenp avatar Oct 14 '24 23:10 guenp

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Oct 14 '24 23:10 CLAassistant

@guenp is attempting to deploy a commit to the Airbyte Growth Team on Vercel.

A member of the Team first needs to authorize it.

vercel[bot] avatar Oct 14 '24 23:10 vercel[bot]

Oops! I didn't mean to add all those labels - I ran the formatter and accidentally pushed files unrelated to this PR. Could someone please remove them? 🙇‍♀️

guenp avatar Oct 18 '24 04:10 guenp

Just wanted to say that this is absolutely epic and both myself and @aaronsteers will absolutely help get this over the finish line.

natikgadzhi avatar Oct 18 '24 23:10 natikgadzhi