airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Make the SerializedDAG & SerializedBaseOp be independent of Task SDK's DAG & BaseOP

Open kaxil opened this issue 6 months ago • 5 comments

Currently, models.BaseOperator and models.DAG use the Task SDK classes.

This was done to maintain backwards-compat and to get 3.0 in a timely manner.

However, we don't want to depend on Task SDK in the Scheduler or the API-server (airflow-core package) - at least not a required dependency.

We only need to ensure we have enough information for the Scheduler & API-server for a DAG & the Operator. This is done via the SerializedDAG & SerializedBaseOp.

So this task involves: figuring out how we can consolidate: SerializedDAG, models.DAG by not inheriting Task SDK; and same for SerializedBaseOp and models.BaseOperator

To do (not ordered by dependency):

  • [x] Timetables
  • [ ] Assets
  • [ ] DAGNode
  • [x] NOTSET and ArgNotSet
  • [ ] DeadlineAlert (and related things?) - Currently WIP
  • [x] BaseOperatorLink
  • [x] ExpandInput
  • [ ] Partitions
  • [ ] Clean up remaining SDK references in Core
  • [ ] Take airflow.serialization.serialized_objects apart
  • [ ] Move MappedOperator to airflow.serialization.definitions

kaxil avatar Jun 24 '25 09:06 kaxil

  • [x] BaseOperator
  • [x] MappedOperator
  • [x] DAG

uranusjr avatar Aug 13 '25 11:08 uranusjr

  • [x] Param
  • [x] ParamsDict
  • [x] TaskGroup

kaxil avatar Aug 31 '25 01:08 kaxil

Not done

uranusjr avatar Oct 21 '25 06:10 uranusjr

@uranusjr we should also add an issue for BaseTrigger/BaseEventTrigger?

amoghrajesh avatar Nov 12 '25 05:11 amoghrajesh

We should eventually, but I’m not sure what to do with triggerer things yet. Let’s put them off for now until we figure triggerer and dag processor split out.

uranusjr avatar Nov 13 '25 10:11 uranusjr