airflow icon indicating copy to clipboard operation
airflow copied to clipboard

WIP - Move dataset dagrun creation to scheduler main

Open dstandish opened this issue 3 years ago • 0 comments

Moves dataset-triggered dagrun creation to the main scheduler loop so that the target dag can be locked.

Makes the run_id of the dataset-triggered dagrun deterministic.

Adds a table to map dataset events to dagruns.

Note: if we decide to go this route, we'll be able to remove querying of dataset-dagrun associations which is done elsewhere (not in this PR yet)

dstandish avatar Jul 11 '22 16:07 dstandish