airflow
airflow copied to clipboard
Enable triggerer queues
Closes: https://github.com/apache/airflow/issues/33818
What?
- Adds optional support for "trigger queues", which allows a given triggerer to consume from a user-configured subset of triggers.
- Adds a
trigger_queuesinstance attribute to the Triggerer, which comes from thetriggerer.consume_trigger_queuesconfig option, or the--consume-trigger-queuesCLI option. - Adds a
trigger_queueinstance attribute to theTrigger, which may either be explicitly set on initialization, or will use the newtriggerer.default_trigger_queueconfig as a default.
- Adds a
Why?
- AIP-67 (multi-tenancy support), to enable team-specific triggerer assignment.
- More generally, and not necessarily specific to multi-tenancy use-cases, to allow for other trigger strategies which may require distinct sets of triggerers.
Remaining TODOs before removing "Draft" status
- [ ] Figure out why alembic is not detecting a change to the
triggertable / not auto-generating a new revision. - [ ] Run db tests once an alembic revision can be generated.
- [ ] Add newsfragment file summarizing this change.
Testing
-
breeze testing core-tests airflow-core/tests/unit --skip-db-tests --use-xdist -
breeze testing providers-tests --run-in-parallel --skip-db-tests -
prek