airflow icon indicating copy to clipboard operation
airflow copied to clipboard

feat: sensor to check status of Dataform action

Open steve148 opened this issue 4 months ago • 2 comments

Summary

Adds a new sensor to the Google Cloud provider that waits on the status of a target in a workflow invocation.

Why might this be useful?

Right now I use Airflow to trigger a Dataform workflow on a schedule. I use the async=True argument on the DataformCreateWorkflowInvocationOperator and the DataformWorkflowInvocationStateSensor to wait until the workflow is complete before running subsequent steps. This approach is simple but comes with some tradeoffs due to its lack of granularity.

  1. If any target in the workflow fails, the sensor will also fail. If I want to have subsequent tasks run for any target that does succeed, this approach will not work.
  2. If I have multiple targets that my subsequent tasks depend on, there may be large gap between when Dataform completes the tasks. This approach has subsequent tasks run when the whole Dataform workflow is complete.

The sensor added in this PR addresses the trade offs listed above by providing a more granular sensor. Instead of waiting for the workflow to complete, it waits for a target within the workflow to complete.

steve148 avatar Oct 15 '24 20:10 steve148