iterative
Refine new/running experiment for plots
When a new experiment is running there are a few entries in the table that affect what is being show in the plot and change their state dynamically. It's important for this logic to be stable (predictable) and not duplicate entities on the plots dashboard, properly pick which experiment is plotted above which, and what colors they have during the run and afterwards.
One of the bugs:
It's not clear why it's stuck (keeps one of the experiments in the middle).
Besides that, on this example:
- we have 4 items selected by running a single experiment (workspace, branch, checkpoint, experiment HEAD). Can we reduce this number?
- we need to check what people see during the run and make sure that at the end there is no jump (e.g. color doesn't change)
- what happens if there are no enough slots (6 selected and I need to run an extra experiments that would enable at least 3 plots)? We should handle this gracefully (Always keep and extra slot for a running experiment in the workspace, don't show it at all?)
One of the bugs:
Caused by https://github.com/iterative/dvc/issues/7693, only a problem with checkpoints.
- we have 4 items selected by running a single experiment (workspace, branch, checkpoint, experiment HEAD). Can we reduce this number?
- we need to check what people see during the run and make sure that at the end there is no jump (e.g. color doesn't change)
- what happens if there are no enough slots (6 selected and I need to run an extra experiments that would enable at least 3 plots)? We should handle this gracefully (Always keep and extra slot for a running experiment in the workspace, don't show it at all?)
The simplest thing to do would be to not auto-select new experiments.
Thanks @mattseddon .
The simplest thing to do would be to not auto-select new experiments.
I don't think this is good from the product perspective unfortunately. We can (and probably we already do?) disable it for the queue.
For the workspace we need to have a clear stable semantics. It should be a single line / images that corresponds to a running experiment (or to the latest checkpoint), we don't want to have 3 or 4 selected things in this case. Plus we need to check for the edge case that I mentioned (experiment's color doesn't jump, what happens if number of selected things > 7 already, etc)
A not insignificant part of this issue seems to have been fixed by 2.34.3 of DVC. We no longer see the issue where a checkpoint experiment will change colors halfway through.
For the workspace we need to have a clear stable semantics. It should be a single line / images that corresponds to a running experiment (or to the latest checkpoint), we don't want to have 3 or 4 selected things in this case. Plus we need to check for the edge case that I mentioned (experiment's color doesn't jump, what happens if number of selected things > 7 already, etc)
Idea:
When a new experiment is created/starts running in the workspace we should auto-select the workspace record. This will provide "live updates" in the plots view for both checkpoint and non-checkpoint experiments (triggered by the workspace watcher - These updates are actually faster than waiting for the experiments table to update and then calling plots diff with the new revision). Once the experiment has finished running we should move the color/selection from the workspace record onto the finished experiment.
Yes, I think it'll be better compared to what we have now. We can try and see how it goes.