pipelines icon indicating copy to clipboard operation
pipelines copied to clipboard

[frontend] ui can't retrieve pod logs when using argo-workflows 3.4+ (`POD_NAMES` change)

Open thesuperzapper opened this issue 1 year ago • 7 comments

Currently, if you upgrade to argo-workflows 3.4+, the log-retrieval function of the ml-pipeline-ui will not work, and the following error will be thrown: Could not get main container logs: Error: Unable to retrieve workflow status: [object Object]

This is because argo-workflows 3.4 changed the default value of their POD_NAMES feature flag to v2, which has changed the way pods in workflows get named.

This means that the frontend will not know the correct name of the pod when it tries to retrieve logs because of our workflowNameFromPodName() function.

https://github.com/kubeflow/pipelines/blob/2c4b128ecec86cdcb3a8aa59c7f7e880efa2556b/frontend/server/workflow-helper.ts#L228-L236

Some docs about POD_NAMES environment variable meaning:

  • https://argoproj.github.io/argo-workflows/environment-variables/#controller
  • https://argoproj.github.io/argo-workflows/environment-variables/#argo-server

Impacted by this bug? Give it a 👍.

thesuperzapper avatar Mar 07 '23 03:03 thesuperzapper

Might be a duplicate of #8449

v-raja avatar Mar 07 '23 09:03 v-raja

/assign @jlyaoyuli, FYI

connor-mccarthy avatar Mar 09 '23 23:03 connor-mccarthy

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Aug 26 '23 07:08 github-actions[bot]

This is still a problem, bumping for bot.

thesuperzapper avatar Aug 27 '23 23:08 thesuperzapper

FWIW, I recently had to make a small change to this in Argo's own UI (see https://github.com/argoproj/argo-workflows/pull/11766), and the code for pod names is limited to this single file.

You can also set POD_NAMES=v1 for compatibility.

agilgur5 avatar Oct 02 '23 01:10 agilgur5

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Dec 31 '23 07:12 github-actions[bot]

Bumping for bot.

thesuperzapper avatar Feb 13 '24 02:02 thesuperzapper

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Apr 13 '24 07:04 github-actions[bot]

@zijianjoy @connor-mccarthy @chensun this is still very important, as while we can tell users to force the old pod naming with the POD_NAMES=v1 env-var, it does make it a lot harder for people to use their Argo Workflows that they already have.

There is a similar issue about how we can't retrieve task results in Argo Workflows 3.4+

  • https://github.com/kubeflow/pipelines/issues/8942

thesuperzapper avatar Apr 13 '24 17:04 thesuperzapper

@thesuperzapper this is resolved in https://github.com/kubeflow/pipelines/pull/10568 via a workaround where we generate the name ourselves, and in future argo releases once this is merged, we can switch to retrieving it from here once again instead

HumairAK avatar Apr 14 '24 18:04 HumairAK