pipelines
pipelines copied to clipboard
[frontend] ui can't retrieve pod logs when using argo-workflows 3.4+ (`POD_NAMES` change)
Currently, if you upgrade to argo-workflows 3.4+, the log-retrieval function of the ml-pipeline-ui
will not work, and the following error will be thrown: Could not get main container logs: Error: Unable to retrieve workflow status: [object Object]
This is because argo-workflows 3.4 changed the default value of their POD_NAMES
feature flag to v2
, which has changed the way pods in workflows get named.
This means that the frontend will not know the correct name of the pod when it tries to retrieve logs because of our workflowNameFromPodName()
function.
https://github.com/kubeflow/pipelines/blob/2c4b128ecec86cdcb3a8aa59c7f7e880efa2556b/frontend/server/workflow-helper.ts#L228-L236
Some docs about POD_NAMES
environment variable meaning:
- https://argoproj.github.io/argo-workflows/environment-variables/#controller
- https://argoproj.github.io/argo-workflows/environment-variables/#argo-server
Impacted by this bug? Give it a 👍.
Might be a duplicate of #8449
/assign @jlyaoyuli, FYI
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
This is still a problem, bumping for bot.
FWIW, I recently had to make a small change to this in Argo's own UI (see https://github.com/argoproj/argo-workflows/pull/11766), and the code for pod names is limited to this single file.
You can also set POD_NAMES=v1
for compatibility.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bumping for bot.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
@zijianjoy @connor-mccarthy @chensun this is still very important, as while we can tell users to force the old pod naming with the POD_NAMES=v1
env-var, it does make it a lot harder for people to use their Argo Workflows that they already have.
There is a similar issue about how we can't retrieve task results in Argo Workflows 3.4+
- https://github.com/kubeflow/pipelines/issues/8942
@thesuperzapper this is resolved in https://github.com/kubeflow/pipelines/pull/10568 via a workaround where we generate the name ourselves, and in future argo releases once this is merged, we can switch to retrieving it from here once again instead