dbt-databricks icon indicating copy to clipboard operation
dbt-databricks copied to clipboard

upgrade databricks-sql-connector to allow 3.2.0 or 3.3.0

Open stephen-bowser opened this issue 1 year ago • 1 comments
trafficstars

Describe the feature

It would be helpful if dbt-databricks was compatible with databricks-sql-connector 3.3.0 or 3.2.0. This would help since these versions are compatible with pyarrow (">=14.0.1,<17"), whereas 3.1.0 is restricted to ^14.0.1

Describe alternatives you've considered

N/A

Additional context

N/A

Who will this benefit?

Anyone who is interested in using the latest versions of pyarrow. In particular, the latest versions (0.19.x) of deltalake require more modern versions of pyarrow than what is currently enabled by dbt-databricks.

stephen-bowser avatar Aug 20 '24 15:08 stephen-bowser

This will come in 1.9; we've become conservative about when we upgrade our dependency versions due to major issues in the migration to 3.0, since users on Databricks Workflows will always get latest available versions if we don't pin them down.

benc-db avatar Aug 20 '24 16:08 benc-db