fix: only cleanup agent pod if exists

[tooptoop4]

Fixes https://github.com/argoproj/argo-workflows/issues/12659

[jswxstw]

The agent pod deletion logic is currently performed in markWorkflowPhase and the PodGC strategy is invalid to the agent pod, so I think it is more appropriate to put it into queuePodsForCleanup. https://github.com/argoproj/argo-workflows/blob/5c8062e55d975aab117e37c7592c0a648a9e9860/workflow/controller/operator.go#L2291 https://github.com/argoproj/argo-workflows/blob/5c8062e55d975aab117e37c7592c0a648a9e9860/workflow/controller/pod_cleanup.go#L13

[jswxstw]

The agent pod deletion logic is currently performed in markWorkflowPhase and the PodGC strategy is invalid to the agent pod, so I think it is more appropriate to put it into queuePodsForCleanup.

https://github.com/argoproj/argo-workflows/blob/5c8062e55d975aab117e37c7592c0a648a9e9860/workflow/controller/operator.go#L2291

https://github.com/argoproj/argo-workflows/blob/5c8062e55d975aab117e37c7592c0a648a9e9860/workflow/controller/pod_cleanup.go#L13

One thing to note is that the agent pod cleanup policy depends on workflowPhase rather than nodePhase.

condition with podGC strategy and workflowPhase	pod cleanup action
(OnPodCompletion || OnPodSuccess || OnWorkflowCompletion) && workflowPhase.Completed()	deletePod
OnWorkflowSuccess && WorkflowSucceeded	deletePod
[Default]workflowPhase.Completed()	labelPodCompleted&&terminateContainers

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity and needs further changes. It will be closed if no further activity occurs.

[github-actions[bot]]

This PR has been closed due to inactivity and lack of changes. If you would like to still work on this PR, please address the review comments and re-open.

[agilgur5]

Superseded by ~#13016~. EDIT: Split to #13294