apache
Backfill Parameters Not Overriding Existing DAG Run Params in Airflow 3
Apache Airflow version
3.1.3
If "Other Airflow 2/3 version" selected, which one?
No response
What happened?
When I run a backfill with parameters on a DAG that already has a DAG run for the same logical date, the parameters from the backfill are not used.
Instead, Airflow reuses the parameters from the existing DAG run (the first run that was created for that logical date). The parameters I specify in the backfill form are ignored.
My use case is that I parallelize work by client: • I have a DAG that runs once per day and can process one specific client based on a parameter (e.g. client_id or data). • I want to be able to backfill only one client for a given day, even if the DAG has already run that day for all clients (or a different client).
Currently, if a DAG run already exists for that date, and I backfill with different parameters, the existing run’s parameters are kept and tasks see the old parameters rather than the new ones.
So effectively: • For new logical dates (no existing DAG run), backfill parameters work as expected. • For existing logical dates, backfill parameters do not override the existing DAG run configuration.
What you think should happen instead?
When I trigger a backfill for a date range and provide parameters (via “Run Parameters” or JSON configuration), the parameters for each DAG run should: • Be taken from what I provide in the backfill form. • Override any existing parameters or configuration for that logical date. • Be visible inside tasks through: • {{ params.my_param }} in templates • context["params"]["my_param"] in Python code • or a typed ParamsDict argument in a task function.
In other words, if a DAG run already exists for a given execution date and I explicitly launch a backfill with new parameters, I expect those new parameters to be applied to that run (or to a new run), so I can effectively “re-run” that date with different params.
This is important for use cases where: • A single DAG processes multiple clients, selected by a parameter. • We need to re-run only one client for a specific date, even if the date already has a DAG run.
How to reproduce
Create the following DAG:
from datetime import datetime
from airflow.models.param import Param, ParamsDict
from airflow.decorators import dag, task
@task
def print_data(params: ParamsDict):
data = params.get("data")
print(f"DATA --> {data}")
@dag(
dag_id='test_backfill_with_params',
start_date=datetime(2023, 1, 1),
params={
"data": Param(
title="My custom data",
type="string",
default="Hello World!",
)
},
schedule='@daily',
catchup=False,
)
def test_backfill_with_params_dag():
print_data()
test_backfill_with_params_dag()
Steps: 1. Go to the Airflow 3 UI. 2. Trigger a manual run for test_backfill_with_params on a given date (e.g. 2025-06-01) with default parameters (or with data = "Hello World!"). 3. Confirm that the task prints DATA --> Hello World!. 4. Now open the Backfill modal for the same DAG. 5. Set a date range that includes the same logical date, for example: • Start: 2025-06-01 00:00 • End: 2025-06-02 00:00 6. In the Run Parameters (or configuration JSON), set: • data = "Overridden Value" 7. Trigger the backfill.
Expected: • For the run corresponding to 2025-06-01, the task should receive params["data"] == "Overridden Value". • The log should show something like: DATA --> Overridden Value
Actual: • For the 2025-06-01 run, the task still sees the original parameter value (e.g. "Hello World!"), or the parameters from the first run that was created for that logical date. • The value provided in the backfill form is ignored for dates that already have a DAG run.
Operating System
Mac OS 14.6 (23G80)
Versions of Apache Airflow Providers
No specific providers involved / default set.
Deployment
Docker-Compose
Deployment details
No response
Anything else?
This behavior makes it difficult to use a single DAG to process multiple clients in parallel via a parameter, because backfills cannot effectively re-run a single client for a day if a run already exists.
It seems that when a DAG run already exists for a logical date, Airflow reuses that run (including its original params) instead of applying the new configuration provided in the backfill request.
Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Code of Conduct
- [x] I agree to follow this project's Code of Conduct
With "all runs" option the existing runs are cleared without passing params https://github.com/apache/airflow/issues/51439#issuecomment-2949835692 .
@dstandish Is it intentional to not pass params to existing dagruns during backfill?
https://github.com/apache/airflow/blob/0a1e798fb71520f5848583f63c006c97c6ea7065/airflow-core/src/airflow/models/backfill.py#L355-L364
I'm also experiencing this, I tried to work around by using the CLI and the "--dag-run-conf" parameter, but I'm seeing the same issue.
