airflow
airflow copied to clipboard
Add on_kill equivalent to Databricks SQL Hook to cancel timed out queries
The Databricks Provider did not implement a mechanism to cancel SQL queries submitted by DatabricksSqlHook. This led to data quality issues, where Airflow would report a cancellation due to timeout; however, the corresponding SQL query would continue to run on Databricks.
This PR uses threading to cancel SQL queries submitted by DatabricksSqlHook.run() once the timeout is exceeded.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst
or {issue_number}.significant.rst
, in newsfragments.