Only enqueue workflows if flytepropeller is leader

[Sovietaced]

Tracking issue

Closes #6435

Why are the changes needed?

Without these changes followers/replicas in highly available deployments will enqueue workflows until they saturate the work queue. Once the work queue is saturated these workflows may become very stale. Once a follower does become a leader it will need to churn through these stale workflows that might not even exist anymore.

What changes were proposed in this pull request?

Added a thread safe bool for whether the propeller is leader and check it before enqueueing workflows.

How was this patch tested?

Not yet but will test in our development environment.

Check all the applicable boxes

• [x] I updated the documentation accordingly.
• [x] All new and existing tests passed.
• [x] All commits are signed-off.

Summary by Bito

This PR implements leader-only workflow enqueuing in HA flytepropeller deployments using an atomic boolean flag, preventing followers from adding stale workflows to the queue. It enhances system reliability and consistency during leadership transitions, and refines workflow handling. The changes also improve logging of workflow phases for failed executions to facilitate better monitoring and debugging.

Unit tests added: False

Estimated effort to review (1-5, lower is better): 1

[flyte-bot]

Code Review Agent Run Status

• Limitations and other issues: ❌ Failure - Bito Automatic Review Skipped - Draft PR
  
  Bito didn't auto-review because this pull request is in draft status.
  To trigger review, mark the PR as ready or type /review in the comment and save.
  You can change draft PR review settings here, or contact the agent instance creator at [email protected].

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.

Project coverage is 58.48%. Comparing base (beb6422) to head (3f88808). Report is 27 commits behind head on master.

Files with missing lines	Patch %	Lines
flytepropeller/pkg/controller/controller.go	0.00%	6 Missing :warning:

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6436   +/-   ##
=======================================
  Coverage   58.47%   58.48%           
=======================================
  Files         940      940           
  Lines       71579    71585    +6     
=======================================
+ Hits        41858    41864    +6     
  Misses      26538    26538           
  Partials     3183     3183           

Flag	Coverage Δ
unittests-datacatalog	59.03% <ø> (ø)
unittests-flyteadmin	56.27% <ø> (+0.02%)	:arrow_up:
unittests-flytecopilot	30.99% <ø> (ø)
unittests-flytectl	64.72% <ø> (ø)
unittests-flyteidl	76.12% <ø> (ø)
unittests-flyteplugins	60.95% <ø> (ø)
unittests-flytepropeller	54.76% <0.00%> (-0.02%)	:arrow_down:
unittests-flytestdlib	64.02% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[flyte-bot]

Code Review Agent Run #99bf3d

Actionable Suggestions - 0

Review Details

• Files reviewed - 1 · Commit Range: b023908..331298e

  • flytepropeller/pkg/controller/controller.go
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful

---

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

• /review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

• Review rules
• General documentation
• FAQ

AI Code Review powered by

[flyte-bot]

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Testing - Test Debug Logging Enhancement	- test_run.py - Added a debug print to log the workflow execution phase, aiding diagnostics during end-to-end tests.
Bug Fix - Leader-based Workflow Enqueuing Fix	- controller.go - Introduced an atomic boolean for leader detection; updated the code to ensure that only the leader enqueues workflows, preventing non-leader replicas from saturating the work queue.

[Sovietaced]

The end2end test is failing but I've tested this in a development environment without issues hmm.

[flyte-bot]

Code Review Agent Run #a53844

Actionable Suggestions - 0

Review Details

• Files reviewed - 1 · Commit Range: 331298e..3f88808

  • boilerplate/flyte/end2end/test_run.py
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful

---

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

• /review - Manually triggers a full AI review.

Refer to the documentation for additional commands.

Configuration

This repository uses code_review_bito You can customize the agent settings here or contact your Bito workspace admin at [email protected].

Documentation & Help

• Review rules
• General documentation
• FAQ

AI Code Review powered by

[Sovietaced]

@machichima this one wasn't ready for review yet since I was trying to work out the build failure. I'll circle back later.