Duplicate Scheduled Flow Runs Triggered in Prefect 2.18

[Hafeez-Aqfer]

Bug summary

When a deployment in Prefect 2.18 is configured with a regular schedule (for example, an hourly IntervalSchedule), the scheduler sometimes emits two flow‐run creation events for the same scheduled timestamp. In practice this looks like:

A flow run is created at the exact top of the interval (e.g. scheduled start time 07:00:00 PM).

Within a few seconds—often 10–20 seconds later—a second flow run is created with that same scheduled start time.

Both runs carry the auto-scheduled tag and point to the same deployment.

Each “duplicate” run has essentially the same scheduled start time (06:03 and 07:03) but a different actual submission timestamp (…:00 vs. …:15, and …:59 vs. …:13), which leads to two runs per interval.

This is causing:

• Wasted resources, since one of the two runs being run for already run task in successful flow.

• Data skew, because downstream tasks re-process the same data input and causes issue.

• Confusion in monitoring, since our dashboards assume exactly one run per interval.

This need to be checked and have some fixes / recommendation. I have attached some screenshots for reference.

Note We cannot yet upgrade to Prefect 3 (see our blocker issue: #15751).

Please suggest some actions on why does the duplicate flow_run even gets scheduled.

We have written internal code to handle this , using read_flow_runs to cancel the flow if there any Running flow_runs. Even the solution turns out to be not working were the response is async.

Version info

Version:             2.18.0
API version:         0.8.4
Python version:      3.12.3
Git commit:          1006d2d8
Built:               Thu, Apr 18, 2024 4:47 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         cloud

Additional context

No response

[nrnavaneet]

Hey, I would like to work on it. Can i be assigned? :)

[lax-vtl]

Hello, we're experiencing the same duplication problem with scheduled flows on our Prefect server.

Version info

Version:        3.4.0
API version:    0.8.4

[jim-ngoo]

Hello, we're experiencing the same duplication problem with scheduled flows on our Prefect server.

Version info

Version:        3.4.0
API version:    0.8.4

I as well encountered the same with v3.4.4

[Noordle]

+1 version 3.4.14

[lax-vtl]

We're experiencing the same issue Prefect v3.4.15

[jeffrose-relay]

We are experiencing this issue on v2.20.19

[idestis]

Any updates on this?

Update:

• on our side, we have used schedule in prefect.yaml, and often flow-runs where duplicated, like same 10:00 to be executed.
• I have changed schedule to schedules and added slug, which solved the issue. Now when we do any kind of redeployment, schedules are not duplicated and properly updated.

[khu-taproot]

We are seeing this in 3.4.24. We define our deployments in prefect.yaml and some of them run on a schedule. After re-deploying some of the deployments have duplicated upcoming runs. If we manually disable the schedule and re-enable it, this clears the runs.

How has this bug been open this long? An orchestrator that can't reliably schedule jobs seems like a breaking change.

[jeffrose-relay]

How has this bug been open this long? An orchestrator that can't reliably schedule jobs seems like a breaking change.

I completely agree. This is absurd that this bug is just sitting here and prefect is not acknowledging anything.

[zzstoatzz]

hi folks, we'll take a look at this issue - sorry for the delay!

I have changed schedule to schedules and added slug, which solved the issue. Now when we do any kind of redeployment, schedules are not duplicated and properly updated.

fwiw my initial reaction here is that this is the root of the issue. if you don't have a slug, the idempotency key can change and you can get duplicates

however, it seems clear that the handling in the case of no slug needs to be clearer / there might be an issue related to multiple scheduler instances (e.g. in Prefect Cloud)

[jim-ngoo]

why close this issue?

Now when we do any kind of redeployment, schedules are not duplicated and properly updated.

is it fixed without explicitly setting slug?

[bunchesofdonald]

@jim-ngoo Closing was unintentional, we're testing a fix at the moment, and our issue tracker can be over zealous.

[S1mpleOW]

i have the same issue

[bunchesofdonald]

A fix for this issue has been deployed to Prefect Cloud and fixes the issue originally seen by @Hafeez-Aqfer.

I'm going to go ahead and close this issue out, but feel free to re-open if you see new duplicate scheduled runs appear. If you do see new duplicates please include the account id, workspace id, and the pair of flow run ids to better help us track down the issue.