Story: Help navigate in a lot of experiments

[maxagin]

This epic is created so we can discuss the concepts together with the DVC team.

Move motivation is here

Ideas

• [x] Star experiments in table #1676
• [ ] Pin experiments to show them at the top
• [ ] Be able to drag and drop rows pinned expS
• [ ] Naming/renaming experiments after execution
• [ ] Able to add context to an experiment (labels https://github.com/iterative/vscode-dvc/issues/1720#issuecomment-1174473366, custom labels, pin, group, comments, free text, notes about experiments etc)
• [x] Collapse exps and checkpoints #2071 & Show a limited amount of checkpoints that can optionally be expanded further #1269

Figma specs

• Star experiments in table
• Label experiments in the table
• Collapse exps and checkpoints
• Group interesting runs in sections with a title
• Able to add context to an experiment (comments)

[shcheklein]

Let repurpose the https://github.com/iterative/vscode-dvc/issues/1676 to "star favorite experiments"? And bring some motivational summary into this epic? wdyt?

[shcheklein]

Or even create a separate ticket and close the #1676

[maxagin]

@shcheklein We can close it any time after. Why not just rename it? It’s still in progress. wdyt?

[shcheklein]

Why not just rename it? It’s still in progress

how would you rename it so that its content is not confusing? (I just don't have a good idea for that tbh).

[maxagin]

how would you rename it

Label/Star experiments ?

[maxagin]

Where? in table Label/Star experiments in table ?

[shcheklein]

@maxagin yes, but that ticket had a lot of general discussion, motivation, ideas, etc. FIrst you would have to expect everyone to read it to get to the bottom, second it overlaps with this one then.

[maxagin]

@shcheklein ok. So how about:

1. Finalize the work
2. Create a new ticked and include only relevant information for further discussion

wdyt?

[maxagin]

Discussion between Ivan and Dave

[shcheklein] Question to answer first here - do we need this on the DVC end. And we need to brainstorm a bit on different possible UX flows that would achieve the same result - easier to navigate.

[dberenbaum] What is the primary use case this intends to target? Is it about having some consistent baseline that you want to save and always compare against? Is it about being able to lock an experiment or two to the top of the table temporarily to see how they stack up against others?

[shcheklein] When I was iterating on get started repo recently I got lost quickly with all the combinations. Partially the reason was that I was changing datasets (and then rerunning the same set of experiments) and we don't show data. But also since dataset name is the same, size in that case was not signaling to me much, etc. Pretty much at some points in the process I felt that I had "good enough" iterations that I can get back if there are no luck with further improvements, but it was impossible to put some label, pin, group, save filter, etc.

To some extent experiment names solve this, but it means you have to think in advance and name each experiment appropriately - it creates a burden by itself.

[dberenbaum]

To some extent experiment names solve this, but it means you have to think in advance and name each experiment appropriately - it creates a burden by itself.

Would naming/renaming experiments after execution help?

In VS Code, it might also make sense to pin experiments in some way to always show them at the top. To me, those types of features might make more sense in VS Code first since it's harder for me to imagine how to make them convenient enough in dvc exp show to be worthwhile. What do you think?

[shcheklein]

Would naming/renaming experiments after execution help?

yep, partially. I would prefer to have some pins/groups/free text form notes about experiments though.

What do you think?

good question. I can tell that CLI is also quite unusable though in such scenarios ... not sure if we want to go very far, but may be at least pin / label experiments in some way?

[maxagin]

Figma specs - Star experiments in table (Phase 1)

[maxagin]

Labels (using GH colour scheme), just so we can keep this ticket alive. Would probably make sense to sort by label, the same way we are planning to show only starred expS.

[maxagin]

Hey @shcheklein . As a part of the task "Help navigate in a lot of experiments" I think that being able to fold not interesting EXPs can be useful. Please let me know WDYT. Concept on Figma

[maxagin]

@shcheklein here is another concept I would like to share in relation to this ticket: "Group interesting runs in sections with the title" Figma

[maxagin]

@mattseddon What do we need to unlock this: Naming/renaming experiments after execution?

[mattseddon]

@mattseddon What do we need to unlock this: Naming/renaming experiments after execution?

I believe the ticket for that is https://github.com/iterative/vscode-dvc/issues/2055

IMO this ticket needs user input and product decisions.

Hard to know what the next steps are from the OP.

[maxagin]

@mattseddon What do we need to unlock this: Naming/renaming experiments after execution?

I believe the ticket for that is https://github.com/iterative/vscode-dvc/issues/2055

Yeah. My question was: How to unlock it? From #2055 "I think it's not very hard to implement, but still need some time to finish it."

[maxagin]

IMO this ticket needs user input and product decisions.

Yes, but it has directions, so I would keep it alive.

[mattseddon]

Yeah. My question was: How to unlock it?

First, we need to decide how high of a priority this is for us. We then need to convey that priority to the DVC team so that they can schedule the work... or not.

[shcheklein]

Let's keep it and get back when we have a bit more user input + we are done with the current activities. I'd like to keep it lightweight as much as possible. It would be a mistake to implement something complicated like labels. On the other hand, we naming + some lightweight notes/comments could be done I think. Anyways, let's not prioritize for now.

[dberenbaum]

My feedback on the experiments table:

• It's not clear which experiments and commits are being shown (or even the relationship between experiments and commits). TBH I'm not sure what's being shown here now so don't know what to suggest 🤔 .
• Sorting and filtering across commits would be helpful (also true for CLI). It's not clear why certain experiments get sorted/filtered and others do not, and it's not helpful to only be able to sort within a commit. Maybe it makes sense to be able to toggle between a Git history view like we have now and more of a flat table view that treats all experiment runs/commits equally?
• Commits without any metrics should probably be hidden/deemphasized.
• What about using the exp show options for filtering the rows to show? It would be more consistent across interfaces and increase flexibility.

[dberenbaum]

Also this is mentioned in #2726, but I would find renaming (as suggested above) helpful. If it's decided it's worth pursuing, we can look into it on the DVC side.

Edit: I think this is a bit different than #2457. No matter what, auto naming is probably not going to be what I would name them manually, and often I don't even know ahead of time what I should name them since my experiments space grows over time and I need to find new ways to differentiate them. Tagging is another useful option here.

[alex000kim]

Modifying the table view to only show differences (vs all available parameters) would help navigate a table with lots of experiments.