`dvclive.log`: log array, tensors and similar objects

[aguschin]

AFAIK, some experiment management libraries (e.g., w&b) can log not only simple python objects, but also more complex, few of them being numpy arrays, torch tensors, tensorflow tensors, etc. It seems like a general functionality which can help a lot of users and make the logging process more convenient for them.

Also, surprisingly to me, w&b client library doesn't have numpy/torch/tensorflow dependencies: https://github.com/wandb/client/blob/master/requirements.txt

Could we implement something similar for the dvclive? If the answer is positive, I'm gonna do a little research and post more examples on what custom types would be great to support.

[dberenbaum]

Logging more data types sounds good. I found this link for wandb:

https://docs.wandb.ai/guides/track/log

Is this what you were thinking? Did you have a specific scenario in mind, and how would you like that data logged?

---

Also, surprisingly to me, w&b client library doesn't have numpy/torch/tensorflow dependencies: https://github.com/wandb/client/blob/master/requirements.txt

They seem to leave it to the user to have the correct dependencies installed, which is similar to the mlflow approach (see https://github.com/iterative/dvclive/issues/25#issuecomment-768623284). Maybe we should have a separate issue to discuss dependency management.

[aguschin]

Is this what you were thinking?

Yes, exactly! The page you found has a great summary. There are a lot of thing W&B can log, I guess few of them fits good in the current functionality, like logging arrays and tensors of length 1. But tracking other types of data would require major extension of the current functionality, I guess. Did we have the similar discussion before somewhere?

you have a specific scenario in mind, and how would you like that data logged?

The specific scenario I first came up with is the following: to log tensors/arrays of length==1 with dvclive, I need to convert them to simple types. To avoid repeating this every time I've written a wrapper, which I'd like to avoid.

More advanced scenario would be to try plotting ROC AUC (TPR vs FPR), or Precision vs Recall, or Precision vs Threshold, or something llke this. To plot this I'd need to store these arrays. If I already use dvclive to store metrics for checkpoints, why not use it to store these arrays to later compare 1st epoch versus 10th epoch versus 100th epoch?

Essentially, user may want to store any kind of artefact over epochs to compare them between each other (few more examples - images with bounding boxes produced by NN, pictures produced by GAN). The question is the same: if I already use dvclive, why not store those artefacts with it?

I can add "checkpoint" keyword to dvc.yaml and then store those artifacts this way, but this is probably a matter of taste. I can imagine I won't use dvclive to avoid code modifications in the first place, but when I already use it, the barrier to make more code modifications becomes lower. If dvclive could substitute some wrapper code one writes to write these artefacts, it can be a good solution to use. Also, this way we can make sure stored artefacts will have out-of-the-box support for dvc plots.

They seem to leave it to the user to have the correct dependencies installed

Looks interesting, thank for the link!

[shcheklein]

More advanced scenario would be to try plotting ROC AUC (TPR vs FPR), or Precision vs Recall, or Precision vs Threshold, or something llke this. To plot this I'd need to store these arrays. If I already use dvclive to store metrics for checkpoints, why not use it to store these arrays to later compare 1st epoch versus 10th epoch versus 100th epoch?

Can W&B or MLFlow do this now? Can they plot multiple ROCs on a single plot?

Essentially, user may want to store any kind of artefact over epochs to compare them between each other

that's correct. But it feels that the most general solution is to output stuff as images.

Otherwise we would have to "teach" DVC or UI or whatever to distinguish different arrays (conf matrix from ROC) and plot them differently. That's what we do with Vega now. Question - would you expect some wizard in Studio/VS Code to do this?

Will DVC plots keep relying on vega templates to show these arrays, etc?

[aguschin]

@shcheklein

Can W&B or MLFlow do this now? Can they plot multiple ROCs on a single plot?

Yes, https://wandb.ai/wandb/plots/reports/Plot-ROC-Curves--VmlldzoyNjk3MDE Also CTRL+F "Roc" on this page: https://docs.wandb.ai/guides/track/log

Question - would you expect some wizard in Studio/VS Code to do this?

It think as a user I would prefer to specify what I store while storing this because then it should be done only once. If we are talking about dvclive, then the closes comparison seems to be like w&b does this:

wandb.log({"roc" : wandb.plot.roc_curve( ground_truth, predictions, labels=None, classes_to_plot=None)})

Then it would be easy to plot this and compare between different branches/experiments, because I don't need to specify this every time.

Per se, W&B has some kind of an image constructor for custom charts: https://docs.wandb.ai/ref/app/features/custom-charts/walkthrough . I didn't use it yet, so I can't tell how frequently it can be useful. It is also based on Vega templates.

[shcheklein]

It think as a user I would prefer to specify what I store while storing this because then it should be done only once. If we are talking about dvclive, then the closes comparison seems to be like w&b does this:

yep, right now it can be done with dvc plots and a predefined vega template and you kinda do it once.

if we decide to add something as highlevel and specific to the plot type to dvclive as wandb has, where would we store this in the repo? how will we connect it with the existing dvc plots show`, etc?

Per se, W&B has some kind of an image constructor for custom charts: https://docs.wandb.ai/ref/app/features/custom-charts/walkthrough . I didn't use it yet, so I can't tell how frequently it can be useful. It is also based on Vega templates.

interesting!

[daavoo]

Essentially, user may want to store any kind of artefact over epochs to compare them between each other (few more examples - images with bounding boxes produced by NN, pictures produced by GAN). The question is the same: if I already use dvclive, why not store those artefacts with it?

I think that the support for this kind of comparison should be first addressed in the "core" dvc (there is an open discussion about it https://github.com/iterative/dvc/discussions/5681) and later on, depending on the outcome, extend dvclive for "logging" images.

[dberenbaum]

The initial example of logging arrays is pretty much already supported in dvc though as plots if I'm understanding correctly, so that is one type for which support could probably be added now in dvclive.

[daavoo]

Created a separate issue for images: https://github.com/iterative/dvclive/issues/150

[daavoo]

The initial example of logging arrays is pretty much already supported in dvc though as plots if I'm understanding correctly, so that is one type for which support could probably be added now in dvclive.

So, if I understand correctly, dvclive would be logging an entire dvc plot (i.e. a full .tsv file) for each call where an array is received?

[dberenbaum]

I don't think much thought has been put into how this would be supported in dvclive. Some options:

1. Each step logging an array/plot could replace the existing file, although this seems problematic without dvc integration since previous steps will be lost.
2. Each step appends to the file, adding a step column. Some changes in core dvc plots would likely be needed to render these plots.
3. Each step writes to a different file path, like plot_step_1.tsv, plot_step_2.tsv. This is easy but gets cluttered and goes against typical dvc conventions.

There are also questions around visualization. Do all steps get shown? Do we need some interactive element to select which steps to show (like https://vega.github.io/vega-lite-v2/docs/bind.html)? If so, would this apply to each chart individually or across all charts like in Studio?

[pared]

I see that I answered some of the questions under #150 Let's keep the discussion here?

I think this issue heavily relies on iterative/dvc#5980 - and we might need to resolve it on dvc side first.

[dberenbaum]

Personally, out of the options above, I see #2 as the most useful.

[dberenbaum]

Another visualization question: Does each step's array show up as a single plot side by side with other steps, or should all steps be shown overlaid on the same plot like dvc plots diff?

Is it even useful to show this data over time, or is it usually more helpful to just show the latest step?

[daavoo]

Personally, out of the options above, I see #2 as the most useful.

I think that using a JSON format would be better than TSV (could be probably used aswel, the JSON structure just feels more intuitive to me) as it kind of aligns with existing functionality (https://dvc.org/doc/command-reference/plots/show#example-hierarchical-data).

So:

dvclive.log("array", [0.96658, 0.97641, 0.87707, 0.87402])

Generates:

{
  "array": [
    { "0": 0.96658},
    { "0": 0.97641 },
    { "0": 0.87707},
    { "0": 0.87402}
  ]
}

And subsequent calls:

dvclive.log("array", [0.8795, 0.88038, 0.89872, 0.87707])

Would update as follows:

{
  "array": [
    { "0": 0.96658, "1": 8795},
    { "0": 0.97641, "1": 88038},
    { "0": 0.87707, "1": 89872},
    { "0": 0.87402, "1": 87707}
  ]
}

Another visualization question: Does each step's array show up as a single plot side by side with other steps, or should all steps be shown overlaid on the same plot like dvc plots diff?

Is it even useful to show this data over time, or is it usually more helpful to just show the latest step?

In the above example, dvc plots show will show the latest step by default. dvc plots show -y {step} could be used to see the plot of a previous step.

For showing steps together I guess we would be discussing something similar to (https://github.com/iterative/dvc/discussions/5980#discussioncomment-1008543) which is essentially supporting multiple y arrays in a single plot.

[daavoo]

Apart from the above discussion, we are currently assuming that logging a 1D array means that the user wants multi-step linear plots.

However, another common use case would be to plot histograms (and thus multi-step histograms) which is not currently supported with default DVC templates.

[dberenbaum]

Not sure I understand what the data is in your example or how it's different from the scalar logging already being done. Is it about logging multiple epochs in one step?

So, if I understand correctly, dvclive would be logging an entire dvc plot (i.e. a full .tsv file) for each call where an array is received?

This is closer to what I thought this issue was about - logging hierarchical or tabular data at each step rather than scalar metrics. This would not assume multi-step linear plots or any particular plot type (although I'm not sure if/how a user could configure the plot through dvclive). Maybe we can start by focusing on how the data gets saved and tackle visualization later.

[pared]

JSON or YAML are definitely more versatile than *sv types. If we want to support logging whole arrays (which makes sense to me if we, for example, generate ROC each step), then we would probably need to abandon dumping into sv.

[daavoo]

Not sure I understand what the data is in your example or how it's different from the scalar logging already being done. Is it about logging multiple epochs in one step?

Sorry for the lack of clarity. It was supposed to be an example of how logging a 1D array would look like, which I thought was the first natural extension of data types to be logged.

So, if I understand correctly, dvclive would be logging an entire dvc plot (i.e. a full .tsv file) for each call where an array is received?

This is closer to what I thought this issue was about - logging hierarchical or tabular data at each step rather than scalar metrics. This would not assume multi-step linear plots or any particular plot type (although I'm not sure if/how a user could configure the plot through dvclive). Maybe we can start by focusing on how the data gets saved and tackle visualization later.

I was trying to show that instead of generating new .tsv files at each step the 1D array could be appended to an existing .json/.yml file, having the key set to step.

[daavoo]

JSON or YAML are definitely more versatile than *sv types. If we want to support logging whole arrays (which makes sense to me if we, for example, generate ROC each step), then we would probably need to abandon dumping into sv.

Each data type could have its own isolated dumping method (part of this P.R.), so won't need to abandon .tsv for scalars, right?

[pared]

Agree, we could possibly extend logging arrays to tsv too, but it would make them a bit unreadable. I have something like this in mind:

step  a
0     {val1}
0     {val2}
1     {val11}
1     {val12}

[dberenbaum]

To me, this is a generalization of #189, where json outputs are saved at each step for plots. Some questions about how to generalize it:

• Does all of this get saved under {dvclive_dir}/{step}? We may want some additional subdir for both this and images, like {dvclive_dir}/artifacts/{step} or something to separate it better from the metrics history TSV files.
• Do we need to support plotting or other visualization of this array-like data or otherwise validate its structure? I think it's useful for now even if DVCLive can't do anything but save the data to file for users to parse later. Can we configure so that these outputs are excluded from plots?

[daavoo]

Does all of this get saved under {dvclive_dir}/{step}? We may want some additional subdir for both this and images, like {dvclive_dir}/artifacts/{step} or something to separate it better from the metrics history TSV files.

In #189 I've implemented:

• Scalars -> Live.log -> Saves to dvclive.dir / scalars
• Images -> Live.log_image -> Saves to dvclive.dir / images
• Plots -> Live.log_plot -> Saves to dvclive.dir / plots

[daavoo]

To me, this is a generalization of #189, where json outputs are saved at each step for plots.

One difference is that, so far in this issue, we have considered logging a separate array to a separate file.

However, the plots in #189 store multiple arrays in a single file. For example, fpr and tpr as part of the roc plot.

[daavoo]

Closing as there were no external requests and we have introduced additional logging methods