`xcom_pull` default behavior differs between Airflow 2.10.2 and 3.0.2 when `task_ids=None`

[MrDXY]

Apache Airflow version

3.0.2

If "Other Airflow 2 version" selected, which one?

2.10.2

What happened?

Summary

The behavior of TaskInstance.xcom_pull() when task_ids=None has changed between Airflow 2.10.2 and 3.0.2, leading to potentially unexpected results and backward compatibility issues.

https://github.com/apache/airflow/blob/79838baef32afab08816bba958b4c1622cf162d6/task-sdk/src/airflow/sdk/execution_time/task_runner.py#L302-L303

Behavior in Airflow 2.10.2

In 2.10.2, calling:

ti.xcom_pull(key="some_key")

would internally delegate to _xcom_pull() and not apply any task_ids filter—effectively pulling the XCom with the given key from any available task, often returning the latest match. This was useful for DAG authors who just wanted to pull a known key from anywhere upstream, especially when only one upstream task returned the value.

https://github.com/apache/airflow/blob/35087d7d10714130cc3e9e9730e34b07fc56938d/airflow/models/taskinstance.py#L610-L617

Behavior in Airflow 3.0.2

In 3.0.2, the implementation changed, and now this call:

ti.xcom_pull(key="some_key") 

defaults to:

task_ids = [self.task_id] 

Which means it only pulls the XCom written by the current task itself, not from any upstream tasks. This is a significant change in default behavior and can silently break DAGs or plugins written for older versions.

https://github.com/apache/airflow/blob/79838baef32afab08816bba958b4c1622cf162d6/task-sdk/src/airflow/sdk/execution_time/task_runner.py#L340-L342

Example Impact In 2.10.2:

# Pulls 'result' from any upstream task that pushed it 
ti.xcom_pull(key="result") 

In 3.0.2:

# Only pulls 'result' from the current task (likely returns None) 
ti.xcom_pull(key="result") 
# Will NOT find upstream result unless task_ids is specified 

What you think should happen instead?

The default behavior should remain consistent across versions to avoid unexpected regressions.

How to reproduce

It's straightforward to reproduce the issue. Define a DAG with two tasks where TaskA runs before TaskB (TaskA >> TaskB), and then:

Have TaskA push an XCom value with key="some_key" and value="some_value".

In TaskB, call xcom_pull("some_key") without specifying task_ids.

In Airflow 2.10.2, this correctly retrieves "some_value" from TaskA. In Airflow 3.0.2, the same call returns None, since it defaults to pulling from TaskB itself.

Operating System

Linux

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• [x] Yes I am willing to submit a PR!

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[amoghrajesh]

Thanks for raising the issue, self-assigning it and will take a look soon.

[jroachgolf84]

Not the exact same issue, but something similar: https://github.com/apache/airflow/issues/51521

[MrDXY]

The map_indexes parameter is affected by the same issue: the recent refactor of Airflow’s xcom_pull() method changed its default behavior when map_indexes is not specified. Previously, it automatically determined whether to return a single value (for non-mapped tasks) or all mapped outputs (for mapped tasks). In the new behavior, it defaults to using self.map_index, which can lead to empty or incorrect results—especially when pulling from a mapped task in a downstream non-mapped task—without any warning or error.

[amoghrajesh]

@MrDXY @jroachgolf84 i believe I have fixed it via: https://github.com/apache/airflow/pull/51568, would you be able to validate it and let me know?

[vatsrahul1001]

@jroachgolf84 @MrDXY let us know if https://github.com/apache/airflow/pull/51568 works for you now

[MrDXY]

Great, Thanks team! I’ll give it a try once I get a free moment—things have been quite busy on my end lately.

[amoghrajesh]

Yeah, do let us know when you get time. BTW, the fix is targeted for 3.1, so the change is on main. You can test there. If you want us to process it, please share your dag if possible and we can check.

[amoghrajesh]

@MrDXY did you get a chance to check this one? We have 3.0.4 released now and 3.0.5 is under vote, could you try again with the new version and open a new issue if this issue persists? Closing this issue with that consensus.

[tvt-devteam]

Seems to still be a problem in Airflow 3.1.2

Many of our xcom_pull calls that previously worked just by calling xcom_pull(key: "some_key") have now started returning None, causing the related dag runs to fail.

Passing in the task_ids explicitly, eg. xcom_pull(key: "some_key", task_ids="some_task_id") seems to fix this.