databricks-sql-python
databricks-sql-python copied to clipboard
databricks
Error Messages with Unity Catalog
On older versions of the connector (version 2.5.2) and connecting to a cluster where unity catalog is not enabled, it would give us the first failure message. However, after updating to the connector (version 2.9.3) and connecting to unity catalog enabled cluster, it's giving us a different error message, which doesn't help figure out what the issue is.
from sqlalchemy import create_engine
# Unity Catalog enabled
uc_eng = create_engine(f"databricks://{CONNECTION_STRING}?catalog=prod", connect_args={'http_path': f'sql/protocolv1/o/{WORKSPACE_ID}/{UC_CLUSTER_ID}'})
uc_eng.execute(f"select * from {TABLE_NAME} limit 1").fetchall() # works
# NOT Unity Catalog ENABLED
non_uc_eng = create_engine(f"databricks://{CONNECTION_STRING}?catalog=prod", connect_args={'http_path': f'sql/protocolv1/o/{WORKSPACE_ID}/{NON_UC_CLUSTER_ID}'})
non_uc_eng.execute("select * from {TABLE_NAME} limit 1").fetchall(). # fails
Failure message:
DatabaseError: (databricks.sql.exc.ServerOperationError) Catalog 'prod' plugin class not found: spark.sql.catalog.prod is not defined
[SQL: select * from prod.{TABLE_NAME} limit 1]
(Background on this error at: http://sqlalche.me/e/13/4xp6)
instead of former:
sqlalchemy.exc.DatabaseError: (databricks.sql.exc.ServerOperationError) [UC_NOT_ENABLED] Unity Catalog is not enabled on this cluster.
