apache
Investigate `Timetable` usage for client-server separation
Analyze current timetable usage in scheduler vs DAG processor. Decide if timetables belong in core, task-sdk, or separate package (airflow-commons/airflow-protocols).
Timetable is kind of the reverse of e.g. dag and asset. It allows customisation, but all the logis is only needed in the scheduler, not the worker nor the dag processor. The main implementation should therefore likely belong in core. However, non-core timetables (e.g. EventsTimetable) may belong in the standard provider instead.
Timetable is kind of the reverse of e.g. dag and asset. It allows customisation, but all the logis is only needed in the scheduler, not the worker nor the dag processor. The main implementation should therefore likely belong in core. However, non-core timetables (e.g.
EventsTimetable) may belong in the standard provider instead.
Doesn't the dag processor use timetable to get the next run time though?
Oh yes, it does, for the first ever run. I forgot about that. This can be changed if we need to though. The scheduler is responsible for calculating all later runs.
Most of the timetable logic need to be in the scheduler. So we should kind of do the opposite of SerializedBaseOperator.
- Existing tiemtable classes in core are scheduler-only.
- Create timetable “stub” classes that only takes arguments without any scheduler logic.
- The stubs are serialised, and deserialised in the scheduler into their “real” timetable class counterparts.
Custom subclasses are more problematic. I guess for now (Airflow 3.1) we can just make that one single subclass fulfill both roles as a stub and real timetable. Maybe at some point we need to have an Airflow Extension SDK for this, and other things plugin-wise that you can register into the scheduler.