metaflow icon indicating copy to clipboard operation
metaflow copied to clipboard

feature: add argo workflows list-runs command

Open saikonen opened this issue 1 year ago • 2 comments

  • add command for listing runs (and filtering by status) for argo-workflows

addresses part of #1387

Note: due to API limitations, this feature required adding a custom label to workflows with the workflow-template as a value, as labels are the only available filter for custom fields on objects. This makes the feature not backwards compatible, as the necessary label does not exist on older runs.

saikonen avatar May 16 '23 13:05 saikonen

Some thoughts and observations on this feature before shipping. The way we currently truncate our deployed Argo workflows names to the maximum that Kubernetes allows for its metadata (253 characters) has some problems, the main one being that these workflows can not be launched through Argo's web UI as it tries to populate a label with the workflow template name.

Creating workflows through Argo web UI has a validation that checks that the metadata name does not exceed 63 characters (so it fits in a label). So Argo is much stricter with naming than what we are with submitted workflows. For the Metaflow use case, we can not start truncating workflow names all the way down to 63 characters though, mainly because of how project branches are named. As an example, these two flows would result in a collision with their workflow template names if truncated to 63 characters:

averylongprojectname.user.verylongfirstname.verylonglastnamecompanyemailaddress.com.FirstFlow
averylongprojectname.user.verylongfirstname.verylonglastnamecompanyemailaddress.com.SecondFlow

which is a very real possibility with the latest changes of allowing email addresses as usernames for deployed flows.

For the scope of this PR, the only solution for adding discoverability to workflows via the Kubernetes API is to introduce a custom label that uses a hash of the workflow template name, as was implemented.

saikonen avatar Jul 05 '23 14:07 saikonen

this feature is somewhat blocked until #1521 is solved. After that we have a canonical way of generating the workflow-template name label, and can use that for finding the runs.

saikonen avatar Sep 04 '23 12:09 saikonen