ibis
ibis copied to clipboard
docs: Missing `databricks` backend docs
Please describe the issue
Hello!
First: thanks for this amazing project :)
I noted that databricks is now enabled as a new backend (#10223, it's also listed under Backend support matrix), but I see no documentation on how to make the connection properly.
Digging into the source code, I noted
- the need for
DATABRICKS_SERVER_HOSTNAME,DATABRICKS_HTTP_PATHandDATABRICKS_TOKENenv vars - the
.connect()needs additional arguments:con = ibis.databricks.connect( catalog="main", schema="default", ) - it relies on Unity Catalog Volumes. At some point of the connection, it tries to create a Volume which is available only in the premium tiers (the need for Unity Catalog). When using a non-premium, I got:
ServerOperationError: [[UC_VOLUMES_NOT_ENABLED](https://docs.microsoft.com/azure/databricks/error-messages/error-classes#uc_volumes_not_enabled)] Support for Unity Catalog Volumes is not enabled on this instance. SQLSTATE: 56038 == SQL == CREATE VOLUME IF NOT EXISTS `root-py=31012-pid=2012` COMMENT 'Ibis memtable storage volume'
It would be awesome if those (and additional) minimal requirements were declared under the "Backends" section of the docs.
I've tested it using ibis-framework==10.2.0, Python 3.10.
Thank you.
Code of Conduct
- [x] I agree to follow this project's Code of Conduct