fix: pod with a restart policy of Never or OnFailure stuck at 'Progressing' (#15317)

[RoelofKuijpers]

This implementation extends the health condition check for pods. Previously the assumption was that Pods with restart policy of Never or OnFailure are hooks with a finite life, these were considered as Progressing instead of Healthy. However, this logic does not apply when the pod is managed by an operator (e.g., Flink operator) and therefore has a restart policy of Never. We introduce a new annotation which existence is checked when the pod is Running, that allows for skipping this logic on restart policy.

[codecov[bot]]

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 47.34%. Comparing base (8849c3f) to head (b216058). :warning: Report is 55 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #709      +/-   ##
==========================================
- Coverage   54.26%   47.34%   -6.93%     
==========================================
  Files          64       64              
  Lines        6164     6537     +373     
==========================================
- Hits         3345     3095     -250     
- Misses       2549     3187     +638     
+ Partials      270      255      -15     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[drewhemm]

This looks like a good approach to the problem.

[drewhemm]

The pod manifest needs the following to pass the tests:

• Compute and storage resources defined
• The alpine tag needs to use something other than latest, e.g. 3.21
• Add automountServiceAccountToken: false to the pod spec, as per the Kubernetes docs

[RoelofKuijpers]

@drewhemm I have made the changes you suggested to get a Quality Gate pass

[drewhemm]

Cool, looks like the last blocking issue is the commit sign off.

[drewhemm]

A non-blocking issue has been flagged by SonarQube, probably best to resolve it as follows:

resources:
  requests:
    ephemeral-storage: "100Mi"

[sonarqubecloud[bot]]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Liammarwood]

This needs to be merged to solve lots of subsequent bugs that have been raised.

[Liammarwood]

@christianh814 would you be able to give this PR a review?

[RoelofKuijpers]

@crenshaw-dev would you be able to give this PR a review? Much appreciated!

[sonarqubecloud[bot]]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[RoelofKuijpers]

Docs required!

Have added info to the documentation now!