databricks
[PECO-1263] Add documentation for execute_async
Note for reviewers: Please review this after reviewing #325.
Description
This pull request adds documentation for the forthcoming execute_async() feature. It describes the interface and gives some usage examples.
Does this method have particular benefits over just running a blocking call in a thread?
Yes, the benefit is to recover a running execution. Imagine you are running a long query (1+ hour) and in the middle, your network cable comes unplugged or your computer runs out of battery. The query continues running on your cluster (incurring cost) but you have no way to recover it or see its results. You can cancel it from the Databricks SQL web UI but apart from that you have no choice but to re-run it and wait another hour for results.
In a second usage scenario, if you deploy the connector in a serverless function or lambda. You can wake the lambda to begin executing the query, commit its handle to state, sleep the lambda for some period of time, and wake up to fetch the results. This isn't possible using threads.