airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Enable triggerer queues

Open zach-overflow opened this issue 1 month ago • 0 comments


Closes: https://github.com/apache/airflow/issues/33818

What?

  • Adds optional support for "trigger queues", which allows a given triggerer to consume from a user-configured subset of triggers.
    • Adds a trigger_queues instance attribute to the Triggerer, which comes from the triggerer.consume_trigger_queues config option, or the --consume-trigger-queues CLI option.
    • Adds a trigger_queue instance attribute to the Trigger, which may either be explicitly set on initialization, or will use the new triggerer.default_trigger_queue config as a default.

Why?

  • AIP-67 (multi-tenancy support), to enable team-specific triggerer assignment.
  • More generally, and not necessarily specific to multi-tenancy use-cases, to allow for other trigger strategies which may require distinct sets of triggerers.

Remaining TODOs before removing "Draft" status

  • [ ] Figure out why alembic is not detecting a change to the trigger table / not auto-generating a new revision.
  • [ ] Run db tests once an alembic revision can be generated.
  • [ ] Add newsfragment file summarizing this change.

Testing

  • breeze testing core-tests airflow-core/tests/unit --skip-db-tests --use-xdist
  • breeze testing providers-tests --run-in-parallel --skip-db-tests
  • prek

zach-overflow avatar Dec 09 '25 14:12 zach-overflow