airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Add on_kill equivalent to Databricks SQL Hook to cancel timed out queries

Open R7L208 opened this issue 5 months ago • 6 comments

The Databricks Provider did not implement a mechanism to cancel SQL queries submitted by DatabricksSqlHook. This led to data quality issues, where Airflow would report a cancellation due to timeout; however, the corresponding SQL query would continue to run on Databricks.

This PR uses threading to cancel SQL queries submitted by DatabricksSqlHook.run() once the timeout is exceeded.


^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

R7L208 avatar Oct 02 '24 14:10 R7L208