apache
Add DatabricksWorkflowPlugin
The DatabricksWorkflowPlugin provides with links in the Airflow UI for tasks that allow us to see the Databricks job run in the Databricks workspace, additionally it also provides link to repair task(s) in the workflow.
Databricks does not allow repairing jobs with single tasks launched
outside the workflow, hence we just provide the link for the job run.
Within the workflow, for each of the task, we provide links to the
job run and repair link for the single task
And at the workflow level, for the job launch task, we provide a
link to repair all failed tasks along with the link for job run in
the Databricks workspace that can be used to monitor the job
in the Databricks account.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
The failing 1 test Provider checks / Compat 2.8.4:P3.8 provider check is unrelated and coming from the Amazon provider tests
=========================== short test summary info ============================
FAILED tests/providers/amazon/aws/auth_manager/test_aws_auth_manager.py::TestAwsAuthManager::test_aws_auth_manager_index - TypeError: is_authorized_custom_view() got an unexpected keyword argument 'fab_action_name'
============ 1 failed, 3059 passed, 5 skipped in 358.28s (0:05:58) =============
@eladkal ?
I will get to it only in few days. Feel free to dismiss my request changes if the concern I raised was handled
I think there is a side-effect of registering plugin that should be removed (so all tests that register plugins should deregister them in setup/teardown - thugh not sure how to do it :)
The failing tests report
FAILED tests/providers/microsoft/azure/operators/test_container_instances.py::TestACIOperator::test_execute_with_spot_discount - AttributeError: 'ContainerGroup' object has no attribute 'priority'
and
FAILED tests/providers/amazon/aws/auth_manager/test_aws_auth_manager.py::TestAwsAuthManager::test_aws_auth_manager_index - TypeError: is_authorized_custom_view() got an unexpected keyword argument 'fab_action_name'
Tried re-running them, but they still fail. They are unrelated to this PR and hence I'm going ahead with merging the PR.