Scheduled trigger misses the next execution date during flow import

[yuri1969]

Describe the issue

We have encountered a rather interesting scheduler behavior during regular operation.

TL;DR

The scheduler does not trigger an execution when the scheduler flow definition gets upgraded via the flow namespace update CLI command issued at the time of the execution.

---

Description

It seems the trigger scheduler gets interrupted by a deployment of new flows. In our environment there are multiple flow definition deployments happening daily via CI/CI using the flow namespace update CLI command.

It is rather unacceptable to miss triggers scheduled to run daily or even weekly in production.

Example

Output of import CLI command issued at 09:53:59,111:

❯ ./kestra-0.20.0-SNAPSHOT flow namespace update 'company.team' '/home/myuser/flow_imports'
2024-10-15 09:53:59,111 INFO  update       i.kestra.core.plugins.PluginScanner Registered 79 core plugins (scan done in 97ms)
101 flow(s) for namespace 'company.team' successfully updated !
- company.team.myflow-37
- company.team.myflow-46
- company.team.myflow-42
...

Execution history of one of the imported flows is missing the execution which would be triggered at 09:54:

---

Reproduction steps

First of all, the reproduction rate not 100% since it relies on timing.

1. Create a test dir flow_imports
2. Create initial test flow definition file flow_imports/myflow.yaml
3. Paste following flow & scheduler definition into the flow_imports/myflow.yaml file:

id: myflow
namespace: company.team

tasks:
  - id: hello
    type: io.kestra.plugin.core.log.Log
    message: Hello World! 🚀

  - id: hello-2
    type: io.kestra.plugin.core.log.Log
    message: Hello World! 🚀

  - id: hello-3
    type: io.kestra.plugin.core.log.Log
    message: Hello World! 🚀

  - id: hello-4
    type: io.kestra.plugin.core.log.Log
    message: Hello World! 🚀

  - id: hello-5
    type: io.kestra.plugin.core.log.Log
    message: Hello World! 🚀

  - id: hello-6
    type: io.kestra.plugin.core.log.Log
    message: Hello World! 🚀

  - id: hello-7
    type: io.kestra.plugin.core.log.Log
    message: Hello World! 🚀

  - id: hello-8
    type: io.kestra.plugin.core.log.Log
    message: Hello World! 🚀

  - id: hello-9
    type: io.kestra.plugin.core.log.Log
    message: Hello World! 🚀

  - id: hello-10
    type: io.kestra.plugin.core.log.Log
    message: Hello World! 🚀

triggers:
  - id: each-min
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "* * * * *"

4. Generate 100 additional copies of the flow (to make the import duration realistic) using the following generator script:

#!/bin/bash

INPUT='myflow.yaml'

if [[ ! -f "$INPUT" ]]; then
    echo "The initial flow file '${INPUT}' not found" >&2
    exit 1
fi

for I in {1..100}; do
    COPY="myflow-${I}.yaml"

    # shellcheck disable=SC2016
    cp "$INPUT" "$COPY" && yq -e -y -i --arg date " $(date '--iso-8601=seconds')" --arg 'id' "myflow-${I}" '.id = $id | .tasks[0].message += $date' "$COPY"
done

5. Start the local Kestra server via ./kestra server local
6. Import all the flow definitions via the ./kestra flow namespace update 'company.team' '<path_to>/flow_imports' CLI command
7. Pick one of the generated flows, such as company.team.myflow-14, and watch its Executions using the UI
8. Update the flow definitions by using the generator script again
9. Since the trigger runs each minute at "00 seconds", wait till ~57th second within the current minute (e.g. 12:13:57 or 10:34:57)
10. Run the import CLI command again
11. In the UI observe whether the picked Execution gets triggered or not

Environment

• Kestra Version: v0.20.0-SNAPSHOT - a95d317
• Operating System (OS/Docker/Kubernetes): Debian
• Java Version (if you don't run kestra in Docker): 21 Temurin