ibis icon indicating copy to clipboard operation
ibis copied to clipboard

docs: Missing `databricks` backend docs

Open filipeo2-mck opened this issue 8 months ago • 0 comments

Please describe the issue

Hello!

First: thanks for this amazing project :)

I noted that databricks is now enabled as a new backend (#10223, it's also listed under Backend support matrix), but I see no documentation on how to make the connection properly.

Digging into the source code, I noted

  • the need for DATABRICKS_SERVER_HOSTNAME, DATABRICKS_HTTP_PATH and DATABRICKS_TOKEN env vars
  • the .connect() needs additional arguments:
    con = ibis.databricks.connect(
        catalog="main",
        schema="default",
    )
    
  • it relies on Unity Catalog Volumes. At some point of the connection, it tries to create a Volume which is available only in the premium tiers (the need for Unity Catalog). When using a non-premium, I got:
    ServerOperationError: [[UC_VOLUMES_NOT_ENABLED](https://docs.microsoft.com/azure/databricks/error-messages/error-classes#uc_volumes_not_enabled)] Support for Unity Catalog Volumes is not enabled on this instance. SQLSTATE: 56038
    == SQL ==
    CREATE VOLUME IF NOT EXISTS `root-py=31012-pid=2012` COMMENT 'Ibis memtable storage volume'
    

It would be awesome if those (and additional) minimal requirements were declared under the "Backends" section of the docs.

I've tested it using ibis-framework==10.2.0, Python 3.10.

Thank you.

Code of Conduct

  • [x] I agree to follow this project's Code of Conduct

filipeo2-mck avatar Mar 10 '25 18:03 filipeo2-mck