plots: How to display images and plots per step

[daavoo]

In the current demo setup, we override the saved image and plot at the end of each step.

Unless using checkpoints, this makes it impossible to visualize images and plots logged at previous steps. And even using checkpoints the current UI doesn't easily allow to check the image and plot from previous steps (requires Apply to workspace and refreshing the view)

In order to enable visualizing images from previous steps with or without checkpoints, in dvclive we implemented the log_image function in a way that images are saved under a step subfolder (https://dvc.org/doc/dvclive/api-reference/live/log_image#step-updates).

However, using log_image in the VSCode UI produces a noisy output, as each step image is displayed in an individual row:

I believe that allowing to display "per-step" images and plots is a relevant scenario for VSCode.

---

In tensorboard, logging an image each step gets rendered using a slider:

---

Related https://github.com/iterative/dvc-render/issues/6

[mattseddon]

It's unlikely that we'll get to this before the release but @maxagin @yalozhkin can you please take a look at this feature request.

[maxagin]

@shcheklein please define the impact and urgency of this ticket.

[shcheklein]

I think it's a good and important improvement. It's not prioritized still. Let's first do plots, onboarding, etc. Everything that we agreed in the Q planning.

[shcheklein]

@daavoo @dberenbaum how do we / do we at all handle this in DVC itself / how about Studio? I think this is similar to the flexible plots, we should come up with a common approach and implement everywhere. Looks like an important change for all the products.

[daavoo]

handle this in DVC itself / how about Studio

We don't handle it in any way in DVC, the HTML is a mess for this scenario.

Images are grouped in a parent folder so at least in VSCode (after https://github.com/iterative/vscode-dvc/issues/1377) and Studio it's easy to hide/show the parent folder from the tree view.

I think we should implement a slider à la TensorBoard, I tried to find something within vega but doesn't look like real a possibility. Not sure if we could have some reusable across UIs, but would be great.

The "tricky" part is how to identify when to render with the slider.

[shcheklein]

yes, what bothers me a bit that this is essentially a new type of plot. It's like metrics vs data series. Here it's an image vs image series. And either we go an opinionated way and hard code some very specific layout or make it general, introduce a plot subtype.

[dberenbaum]

@shcheklein I think it's related to https://github.com/iterative/dvclive/issues/326. If we enforce a DVC approach of keeping one image per step, it should simplify it. I think it's enough for now to be able to compare images across steps/commits like we diff other outputs.

Edit:

To summarize, IMO it makes sense to:

1. Keep one image per step at the same file path and overwrite that path at each step.
2. Use the slider to diff that path across steps/commits.

[daavoo]

@dberenbaum Without checkpoints, how would the UI (DVC CLI / VSCode / Studio) access the overwritten images?

[dberenbaum]

It would have to depend on checkpoints. That might be okay if we work on improving/making easier the checkpoints workflow and funnel users to it.

Edit: The alternative seems to be supporting both commit-based diffs and filepath-based diffs, and that seems more confusing and complicated to me (both to implement and explain).