`dvclive.log`: log array, tensors and similar objects

[aguschin]

AFAIK, some experiment management libraries (e.g., w&b) can log not only simple python objects, but also more complex, few of them being numpy arrays, torch tensors, tensorflow tensors, etc. It seems like a general functionality which can help a lot of users and make the logging process more convenient for them.

Also, surprisingly to me, w&b client library doesn't have numpy/torch/tensorflow dependencies: https://github.com/wandb/client/blob/master/requirements.txt

Could we implement something similar for the dvclive? If the answer is positive, I'm gonna do a little research and post more examples on what custom types would be great to support.

[dberenbaum]

Logging more data types sounds good. I found this link for wandb:

https://docs.wandb.ai/guides/track/log

Is this what you were thinking? Did you have a specific scenario in mind, and how would you like that data logged?

---

Also, surprisingly to me, w&b client library doesn't have numpy/torch/tensorflow dependencies: https://github.com/wandb/client/blob/master/requirements.txt

They seem to leave it to the user to have the correct dependencies installed, which is similar to the mlflow approach (see #25 (comment)). Maybe we should have a separate issue to discuss dependency management.

[aguschin]

Is this what you were thinking?

Yes, exactly! The page you found has a great summary. There are a lot of thing W&B can log, I guess few of them fits good in the current functionality, like logging arrays and tensors of length 1. But tracking other types of data would require major extension of the current functionality, I guess. Did we have the similar discussion before somewhere?

you have a specific scenario in mind, and how would you like that data logged?

The specific scenario I first came up with is the following: to log tensors/arrays of length==1 with dvclive, I need to convert them to simple types. To avoid repeating this every time I've written a wrapper, which I'd like to avoid.

More advanced scenario would be to try plotting ROC AUC (TPR vs FPR), or Precision vs Recall, or Precision vs Threshold, or something llke this. To plot this I'd need to store these arrays. If I already use dvclive to store metrics for checkpoints, why not use it to store these arrays to later compare 1st epoch versus 10th epoch versus 100th epoch?

Essentially, user may want to store any kind of artefact over epochs to compare them between each other (few more examples - images with bounding boxes produced by NN, pictures produced by GAN). The question is the same: if I already use dvclive, why not store those artefacts with it?

I can add "checkpoint" keyword to dvc.yaml and then store those artifacts this way, but this is probably a matter of taste. I can imagine I won't use dvclive to avoid code modifications in the first place, but when I already use it, the barrier to make more code modifications becomes lower. If dvclive could substitute some wrapper code one writes to write these artefacts, it can be a good solution to use. Also, this way we can make sure stored artefacts will have out-of-the-box support for dvc plots.

They seem to leave it to the user to have the correct dependencies installed

Looks interesting, thank for the link!

[shcheklein]

More advanced scenario would be to try plotting ROC AUC (TPR vs FPR), or Precision vs Recall, or Precision vs Threshold, or something llke this. To plot this I'd need to store these arrays. If I already use dvclive to store metrics for checkpoints, why not use it to store these arrays to later compare 1st epoch versus 10th epoch versus 100th epoch?

Can W&B or MLFlow do this now? Can they plot multiple ROCs on a single plot?

Essentially, user may want to store any kind of artefact over epochs to compare them between each other

that's correct. But it feels that the most general solution is to output stuff as images.

Otherwise we would have to "teach" DVC or UI or whatever to distinguish different arrays (conf matrix from ROC) and plot them differently. That's what we do with Vega now. Question - would you expect some wizard in Studio/VS Code to do this?

Will DVC plots keep relying on vega templates to show these arrays, etc?

[aguschin]

@shcheklein

Can W&B or MLFlow do this now? Can they plot multiple ROCs on a single plot?

Yes, https://wandb.ai/wandb/plots/reports/Plot-ROC-Curves--VmlldzoyNjk3MDE
Also CTRL+F "Roc" on this page: https://docs.wandb.ai/guides/track/log

Question - would you expect some wizard in Studio/VS Code to do this?

It think as a user I would prefer to specify what I store while storing this because then it should be done only once. If we are talking about dvclive, then the closes comparison seems to be like w&b does this:

wandb.log({"roc" : wandb.plot.roc_curve( ground_truth, predictions, labels=None, classes_to_plot=None)})

Then it would be easy to plot this and compare between different branches/experiments, because I don't need to specify this every time.

Per se, W&B has some kind of an image constructor for custom charts: https://docs.wandb.ai/ref/app/features/custom-charts/walkthrough . I didn't use it yet, so I can't tell how frequently it can be useful. It is also based on Vega templates.

[shcheklein]

It think as a user I would prefer to specify what I store while storing this because then it should be done only once. If we are talking about dvclive, then the closes comparison seems to be like w&b does this:

yep, right now it can be done with dvc plots and a predefined vega template and you kinda do it once.

if we decide to add something as highlevel and specific to the plot type to dvclive as wandb has, where would we store this in the repo? how will we connect it with the existing dvc plots show`, etc?

Per se, W&B has some kind of an image constructor for custom charts: https://docs.wandb.ai/ref/app/features/custom-charts/walkthrough . I didn't use it yet, so I can't tell how frequently it can be useful. It is also based on Vega templates.

interesting!

[daavoo]

Essentially, user may want to store any kind of artefact over epochs to compare them between each other (few more examples - images with bounding boxes produced by NN, pictures produced by GAN). The question is the same: if I already use dvclive, why not store those artefacts with it?

I think that the support for this kind of comparison should be first addressed in the "core" dvc (there is an open discussion about it iterative/dvc#5681) and later on, depending on the outcome, extend dvclive for "logging" images.

[dberenbaum]

The initial example of logging arrays is pretty much already supported in dvc though as plots if I'm understanding correctly, so that is one type for which support could probably be added now in dvclive.

[daavoo]

Created a separate issue for images: #150

[daavoo]

The initial example of logging arrays is pretty much already supported in dvc though as plots if I'm understanding correctly, so that is one type for which support could probably be added now in dvclive.

So, if I understand correctly, dvclive would be logging an entire dvc plot (i.e. a full .tsv file) for each call where an array is received?

[dberenbaum]

I don't think much thought has been put into how this would be supported in dvclive. Some options:

1. Each step logging an array/plot could replace the existing file, although this seems problematic without dvc integration since previous steps will be lost.
2. Each step appends to the file, adding a step column. Some changes in core dvc plots would likely be needed to render these plots.
3. Each step writes to a different file path, like plot_step_1.tsv, plot_step_2.tsv. This is easy but gets cluttered and goes against typical dvc conventions.

There are also questions around visualization. Do all steps get shown? Do we need some interactive element to select which steps to show (like https://vega.github.io/vega-lite-v2/docs/bind.html)? If so, would this apply to each chart individually or across all charts like in Studio?

[pared]

I see that I answered some of the questions under #150
Let's keep the discussion here?

I think this issue heavily relies on iterative/dvc#5980 - and we might need to resolve it on dvc side first.