airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Have `BaseOperator` get DAG from current task group if necessary

Open sean-rose opened this issue 2 years ago • 6 comments

If there isn't an active DAG context but there is an active task group context where the task group has an associated DAG, then currently operators won't get the DAG from the task group, so we either have to explicitly trigger a new DAG context:

# Assume `dag` has been defined elsewhere without using a DAG context.
with dag:
    with TaskGroup(...):
        task = Operator(...)

or manually pass the DAG into every operator:

# Assume `dag` has been defined elsewhere without using a DAG context.
with TaskGroup(..., dag=dag):
    task = Operator(..., dag=dag)

This PR changes BaseOperator to get the DAG from the current task group if it hasn't been manually passed in or found via an active DAG context:

# Assume `dag` has been defined elsewhere without using a DAG context.
with TaskGroup(..., dag=dag):
    task = Operator(...)
    assert task.dag == dag

^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

sean-rose avatar Nov 29 '23 23:11 sean-rose

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Jan 14 '24 00:01 github-actions[bot]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Mar 03 '24 00:03 github-actions[bot]

This sounds reasonable to me. Although the dag=dag approach is generally disfavoured nowadays, and you can always do

with dag, TaskGroup(...):
    ...

instead, fetching the DAG from the TaskGroup context seems like the right thing to do.

The tests need to be fixed though.

uranusjr avatar Mar 04 '24 21:03 uranusjr

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Apr 19 '24 00:04 github-actions[bot]

@sean-rose can you fix the tests?

eladkal avatar Apr 21 '24 06:04 eladkal

@sean-rose can you fix the tests?

Sure, I'll try to find some time soon to take a crack at that.

sean-rose avatar Apr 23 '24 22:04 sean-rose

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Aug 31 '24 06:08 github-actions[bot]