[Feature] Group executions by labels

[katrogan]

Motivation: Why do you think this is important? Users can provide labels for workflow executions using launch plans at execution time. It's useful to be able to search across executions by label.

Goal: What should the final outcome look like, ideally? There should be an endpoint exposed for searching executions by label and the UI should support search as well.

Describe alternatives you've considered Current behavior which exposes labels in protobuf only and is retrievable for each individual execution

Flyte component

• [ ] Overall
• [ ] Flyte Setup and Installation scripts
• [ ] Flyte Documentation
• [ ] Flyte communication (slack/email etc)
• [ ] FlytePropeller
• [ ] FlyteIDL (Flyte specification language)
• [ ] Flytekit (Python SDK)
• [x] FlyteAdmin (Control Plane service)
• [ ] FlytePlugins
• [ ] DataCatalog
• [ ] FlyteStdlib (common libraries)
• [x] FlyteConsole (UI)
• [ ] Other

[Optional] Propose: Link/Inline

Additional context

Is this a blocker for you to adopt Flyte Please let us know if this makes it impossible to adopt Flyte

[jeevb]

I think this would come in handy for teams building platforms on top of Flyte, specifically allowing users to filter/list executions with a rich set of tags (e.g. user_id, git_branch, experiment_id, etc.). project might be overkill for distinguishing between executions, but tags could be ideal!

[kumare3]

@jeevb do you have an idea of priority for this task - when do you guys need it?

[katrogan]

hey @jeevb just fyi you can actually now filter by user_id if you're on the latest deployment of flyte/flyteadmin

[kumare3]

@jeevb any interest in driving this spec?

[jeevb]

Pinging relevant folks: @tvetterli-fn @MarvinBertin. What do y'all think?

[asahalyft]

I requested for this feature in our internal channel with Flyte team. Our Business Use Case is as follows:

We create Flyte Workflows for Model Monitoring. We want to make Model Monitoring mandatory. Before we deploy any model for serving we want to make sure that there exists a workflow to monitor the model output against some ground truth. There may not be a workflow execution yet, which is fine, the user may be deploying the model just today, but there must exist a Flyte workflow.

We want to see if there are some workflows registered by a given name or some metadata or is there a provision to attach special tags while creating the workflows.

e.g. we want to add some custom tags like {model_id: 12229, model_version:v1, team: pricing, some_key:some_value} to the workflows apart from project name and the mandatory attributes - just like we can provide tags for aws objects and then search for those aws objects by those tags.

Once we are sure that a Workflow exists then we allow the model to be deployed for serving.

[katrogan]

thanks @asahalyft for the background! as part of https://github.com/flyteorg/flyte/issues/531 we want to expand the domain of what's searchable to include descriptions, labels, tags and more.

cc @mayitbeegh

[kumare3]

I think we should prioritize this feature as there are some fantastic use cases that are emerging. cc @jsonporter / @andrewwdye / @eapolinario / @cosmicBboy / @EngHabu