databricks
[Feature Request] Support async execution
We are currently using https://github.com/databricks/databricks-sql-python/blob/a5b1ab0745bb0a4e917ea800f36ae4b74d079a75/examples/README.md?plain=1#L34 to fetch statement ID when executing queries using sqlalchemy. However, the UUID is not returned at the very end of the query execution. And if the query fails, it never returns the statement ID. Is it possible to support async execution so that we can get the statement ID asap.
Is there a timeline for getting proper async support? We would love that to speed up our Databricks workflows! There was clearly some work going on by susodapop (wo seems to have left Databricks) in #322 , #325 and Branch peco-1263-staging -- all of which has stalled 7 months ago.
+1 this affects databricks usability in the external systems as it is not possible to navigate to the query when it is running, failed or times out and makes debugging much more complex than it should be
@rcypher-databricks @yunbodeng-db @andrefurlan-db @jackyhu-db @benc-db @kravets-levko
There is a community desire for async execution for various reasons. #82 and #176
My use case is OpenAI parallel tool calling which can generate multiple queries, but are executed sequentially due to lack of async support.
I don't feel like the API is appropriately async. When running a Merge-Statement the call to Cursor.execute_async will wait for the Statement to complete, only then returning to my Python code. I have traced this back to the inclusion of getDirectResults=ttypes.TSparkGetDirectResults(...) in ThriftBackend.executeCommand. Removing this parameter makes the execution actually run async. I have had no luck with changing maxRows or maxBytes (neither 0 nor 1)
@MeinAccount We have fixed the issue and it is available in the latest release of the PySQL Connector v3.7.3
So is it possible to execute a query using await, are there query functions that are defined with async def ...? Having native python async methods would be really helpful towards integrating properly with the rest of the Python ecosystem
