airflow
airflow copied to clipboard
WIP - Move dataset dagrun creation to scheduler main
Moves dataset-triggered dagrun creation to the main scheduler loop so that the target dag can be locked.
Makes the run_id of the dataset-triggered dagrun deterministic.
Adds a table to map dataset events to dagruns.
Note: if we decide to go this route, we'll be able to remove querying of dataset-dagrun associations which is done elsewhere (not in this PR yet)