[Core feature] Task retry support in Flyte Connectors

[machichima]

Tracking issue

Part of #6465

Why are the changes needed?

Currently, our Connector cannot retry when task fail, which results in a loss of features for end users migrating from a plugin to a connector

What changes were proposed in this pull request?

use PhaseInfoRetryableFailure when task execution fail.

How was this patch tested?

Create a dummy connector that keep returning RETRYABLE_FAILED. Set the default-max-attempt = 3:

propeller:
  ...
  node-config:  
    default-max-attempts: 3

Run the dummy connector task and confirm it attempt 3 times:

Labels

Please add one or more of the following labels to categorize your PR:

• added: For new features.
• changed: For changes in existing functionality.
• deprecated: For soon-to-be-removed features.
• removed: For features being removed.
• fixed: For any bug fixed.
• security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

• [ ] I updated the documentation accordingly.
• [x] All new and existing tests passed.
• [x] All commits are signed-off.

Related PRs

https://github.com/flyteorg/flytekit/pull/3265

Docs link

Summary by Bito

This pull request enhances Flyte Connectors by introducing a retry mechanism for tasks that fail due to retryable errors, adding a new execution phase 'RETRYABLE_FAILED'. This improvement ensures tasks can attempt retries, addressing critical functionality gaps for users migrating from plugins to connectors and improving reliability and user experience.

[flyte-bot]

Bito Automatic Review Skipped - Draft PR

Bito didn't auto-review because this pull request is in draft status.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change draft PR review settings here, or contact your Bito workspace admin at [email protected].

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 58.49%. Comparing base (c710fb6) to head (c98344e). Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
...yteplugins/go/tasks/plugins/webapi/agent/plugin.go	0.00%	2 Missing :warning:
...lugins/go/tasks/plugins/webapi/connector/plugin.go	0.00%	2 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6486      +/-   ##
==========================================
- Coverage   58.50%   58.49%   -0.02%     
==========================================
  Files         940      940              
  Lines       71582    71586       +4     
==========================================
- Hits        41882    41875       -7     
- Misses      26520    26531      +11     
  Partials     3180     3180              

Flag	Coverage Δ
unittests-datacatalog	59.03% <ø> (ø)
unittests-flyteadmin	56.22% <ø> (-0.03%)	:arrow_down:
unittests-flytecopilot	30.99% <ø> (ø)
unittests-flytectl	64.72% <ø> (ø)
unittests-flyteidl	76.12% <ø> (ø)
unittests-flyteplugins	61.06% <0.00%> (-0.02%)	:arrow_down:
unittests-flytepropeller	54.78% <ø> (ø)
unittests-flytestdlib	64.02% <ø> (-0.02%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[flyte-bot]

Bito Automatic Review Skipped - Draft PR

Bito didn't auto-review because this pull request is in draft status.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change draft PR review settings here, or contact your Bito workspace admin at [email protected].

[flyte-bot]

Bito Automatic Review Skipped - Draft PR

Bito didn't auto-review because this pull request is in draft status.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change draft PR review settings here, or contact your Bito workspace admin at [email protected].