bdit_data-sources icon indicating copy to clipboard operation
bdit_data-sources copied to clipboard

#1070 use `pre_execute` to reduce # of tasks in partition create taskgroups

Open gabrielwol opened this issue 5 months ago • 0 comments

What this pull request accomplishes:

  • switch from using @task.short_circuit operators to pre_execute = reduces # of tasks for partition creates.
    • Originally intended to use @task.run_if which would have had the same impact (fewer tasks), but the partition creates are currently PostgresOperators which would have required refactoring them to @task operators.

Tested by re-running today's miovision_pull DAG ✅

Issue(s) this solves:

  • #1070

What, in particular, needs to reviewed:

  • test_dags action is failing on deprecated pull_wys and pull_miovision... Should we delete these DAGs once and for all?

What needs to be done by a sysadmin after this PR is merged

E.g.: these tables need to be migrated/created in the production schema.

gabrielwol avatar Sep 10 '24 18:09 gabrielwol