Update DuckDB connector for destinations V2
What
Enable Destinations V2 support for the DuckDB connector
How
- Migrated a few files (constants.py, exceptions.py, name_normalizers.py and types.py) from the pyairbyte repo to airbyte-cdk/python/airbyte_cdk/sql
- Added new
_get_sql_column_definitionsmethod toDestinationDuckDbbased on pyairbyte'sSqlProcessorBase._get_sql_column_definitions - Add SQLAlchemy and rich dependencies
- Moved all tests into tests/ folder to make it easier to set up VS Code
Review guide
- airbyte-cdk/python/airbyte_cdk/sql/constants.py: copied from PyAirbyte
- airbyte-cdk/python/airbyte_cdk/sql/exceptions.py: copied from PyAirbyte
- airbyte-cdk/python/airbyte_cdk/sql/name_normalizers.py: copied from PyAirbyte
- airbyte-cdk/python/airbyte_cdk/sql/types.py: copied from PyAirbyte
- airbyte-integrations/connectors/destination-duckdb/destination_duckdb/destination.py: added
_get_sql_column_definitionsand various other changes
User Impact
Airbyte will no longer create tables with column names _airbyte_ab_id, _airbyte_emitted_at, _airbyte_data, but instead _airbyte_raw_id, _airbyte_extracted_at, _airbyte_meta plus any columns defined in the JSON schema.
Can this PR be safely reverted and rolled back?
- [x] YES 💚
- [ ] NO ❌
@guenp is attempting to deploy a commit to the Airbyte Growth Team on Vercel.
A member of the Team first needs to authorize it.
Oops! I didn't mean to add all those labels - I ran the formatter and accidentally pushed files unrelated to this PR. Could someone please remove them? 🙇♀️
Just wanted to say that this is absolutely epic and both myself and @aaronsteers will absolutely help get this over the finish line.