Databricks-SDK Version <0.9 for Dagster-Databricks Package

[Zeulni]

Dagster version

1.7.1

What's the issue?

For the current Dagster version (i.e., 1.7.1), the databricks-sdk version required for the dagster-databricks package is <0.9 (see here), although version 0.25.1 is already available. This caused some problems with our DBT setup, as the newest version of the dbt-databricks package requires a databricks-sdk version of 0.17.0 (see here). Thus, installing these two packages in the same environment is not possible without a version conflict.

To prevent this, we had to fork the dagster-databricks package, adapt the databricks-sdk version to 0.17.0 in the setup.py and pull the updated package from our own package registry. We also updated the databricks-sdk version on the Databricks cluster to 0.17.0.

What did you expect to happen?

When running the pipelines with the databricks-sdk version 0.17.0 (locally and on the Databricks cluster), I expected our pipelines to fail, as we intentionally used a higher version than specified in the dagster-databricks setup.py. However all our pipelines seem to work as expected. So we were wondering, why the databricks-sdk version is required to be so low for the dagster-databricks package? Is it only for certain features, which we might not use in our case?

Any help would be appreciated, since this caused lots of dependency problems in our project.

How to reproduce?

We are using Dagster 1.7.1, dagster-databricks 0.23.1 and the databricks-pyspark-step-launcher.

Deployment type

Local

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[lamalex]

I just submitted a PR for this! Please give my branch a try if you can!

[Zeulni]

Hi @lamalex, thanks for your PR! I saw you bumped the databricks-sdk version in the setup.py. We did the same and this worked for us!