brickflow icon indicating copy to clipboard operation
brickflow copied to clipboard

[FEATURE] TaskDependencySensor flexibility Airflow sensor

Open IMC07 opened this issue 5 months ago • 1 comments

Is your feature request related to a problem? Please describe. Given the fact that in the taskdependencysensor we can only filter the API response on greater then or equal to timestamp-x, the workflow we make is non-deterministic, because when we would rerun a pipeline, the window of time changes because the end-time is 'now'. We would like to be able to make this deterministic, to really be able to see if there was a succes within the timeframe we allow for our pipeline.

Cloud Information

  • [x] AWS
  • [ ] Azure
  • [ ] GCP
  • [ ] Other

Describe the solution you'd like I have created a simple, slightly ugly solution in order to stay backwards compatible, see PR. However, if you agree, I would like to refactor the sensor to allow for more flexibility on filtering results from the API.

Describe alternatives you've considered Create a more flexible API filter as said above.

Additional context Add any other context or screenshots about the feature request here.

IMC07 avatar Sep 12 '24 13:09 IMC07