bdit_data-sources
bdit_data-sources copied to clipboard
#1070 use `pre_execute` to reduce # of tasks in partition create taskgroups
What this pull request accomplishes:
- switch from using
@task.short_circuit
operators topre_execute
= reduces # of tasks for partition creates.- Originally intended to use
@task.run_if
which would have had the same impact (fewer tasks), but the partition creates are currently PostgresOperators which would have required refactoring them to @task operators.
- Originally intended to use
Tested by re-running today's miovision_pull
DAG ✅
Issue(s) this solves:
- #1070
What, in particular, needs to reviewed:
-
test_dags
action is failing on deprecatedpull_wys
andpull_miovision
... Should we delete these DAGs once and for all?
What needs to be done by a sysadmin after this PR is merged
E.g.: these tables need to be migrated/created in the production schema.