Metrics: separating scalar and continuous/graphs metrics

[dmpetrov]

Problem

There are two types of metrics in ML projects: scalar metrics or just a number (like AUC) and continuous metrics or a sequence of numbers (like ROC curve - an array of numbers).

Today we support csv and json files as metrics without specifying what metrics are needed. In fact, DVC commands support only scalars: dvc metrics diff.

Solution

It is important to support both, understand the semantics of the metrics and provide more value.

1. For scalar metrics, DVC should provide (and it does provide it now) dvc metrics diff functionality. We just need clearly file format in the documentation to make sure only scalar metrics are supported. Json/Ini are both good formats for scalar metrics.
2. For continuous metrics, DVC should provide plots/graph visualization functionality. Like dvc viz show roc.json and dvc viz diff roc.json HEAD HEAD^^. Both of the commands should generate graphs. The second command - two graphs (not diff).

Types

Types might improve visualization a lot. There many possible types of graphs for dvc viz. It would be great to support a few common ones like regular plot, confusion matrix.

For scalar types, it is also important to specify a type. A scalar can be the result of the minimization of some function or maximization. If the user can specify this information it might help to show proper color: red if max meterics was decreased and green if increased.

We should not make the metrics file format complicated. So, it might be a better solution to specify metrics format in a separate file (like Dvcfile or special metrics file).

- scalars_types:
      AUC: max
      error_1: min
- metrics:
      roc.json: plot
      error.csv: plot
      mx.json: confusion_matrix

Open questions

1. Should we include dvc viz in the core dvc package? Problem - most likely it will bring dependencies to some graphics libraries which can cause installation issues in OS without graphics (like default EC2 instances). What options do we have?
2. How we should specify the format of the graphs. Do we need any customization? For example, mlflow supports only one type of graph and cannot show something more advance (like a confusion matrix). If we choose Vega-graph as the format we can support graph customization.
3. The metrics can be related to experiment parameters (see Introduce hyper parameters and config #3393) because we are introducing new DVC file types and can use a single DVC file to preserve some information about metrics or metrics types.

Actions

• Modify DVC documentation - only scalar metrics are supported in the current format and DVC commands.
• Implement continuous metrics with dvc viz command.
• Add support of types.

[efiop]

From discussion: need to delete/hide -m option for dvc run so that metrics are in git. dvc metrics add stays though.

[dmpetrov]

In order to make this scalar/continuous metrics separation, the current scalar metrics need to be redesigned.

1. CSV/TSV types are specific to continuous metrics. There is no need to keep them in current, scalar ones. Let's get rid of CSV/TSV/HCSV/HTSV metrics format.
2. Add support for YAML since it is an often-used format - Introduce hyper parameters and config #3393 (comment)
3. With two simple json/yaml formats, do we really need raw format? Let's get rid of raw metrics format.
4. Do we really need xpath in dvc-files? I see a clear use case in metrics show as a filter but not in other commands. Let's consider removing xpath.
5. Metrics type was mostly used for distinguishing xpath algorithms between hierarchical json and flat csv/tsv/hcsv/htsv. It looks like it is not needed if only json/yaml left. Let's remove metrics type.
6. Change docs - we should be very explicit about supporting only JSON/YAML format.
7. [Optional] Redesign metrics show. Transform it into a table-like format.

[jorgeorpinel]

continuous/graphs

I would call these "series", "array", or "track" metrics. Not sure about term "continuous". Just a thought

Json/Ini are both good formats for scalar metrics.

ini=YAML?

Should we include dvc viz in the core dvc package?

Probably not. DVCx then? In this case x can also mean "eXperiment mgmt".

I have doubts about the command names though. So we're keeping dvc metrics only for scalars which have an entry called scalars_types in the "Metricsfile", and new command dvc viz for series with entry metrics? I think we can improve the naming, I'll come up with a suggestion 🙂

If we choose Vega-graph as the format we can support graph customization.

Sounds smart to leave this to a viz package like thar and just refer users to their formatting guidelines.

[jorgeorpinel]

metrics can be related to experiment parameters (see #3393) because we are introducing new DVC file types and can use a single DVC file to preserve some information about metrics

Is there a decision on what the hyperparam metafiles are going to look like? It could be relevant to know for this point ^

do we really need raw format? Let's get rid of raw metrics format

What about metrics that just spit a single number into a text file. That's not really something we want to support? (It could even be stored directly in the DVC-file in the future.)

[jorgeorpinel]

Note: point 6. in #3409 (comment) above should be automatic with our process. It will be addressed in iterative/dvc.org/pull/1097 and in the docs PR derived from addressing 2. 🙂

[jorgeorpinel]

@efiop

From discussion: need to delete/hide -m option

So we would remove --metrics as well and keep --metrics-no-cache? I like the shorter option name better. How about doing so but renaming --metrics-no-cache -> --metric (singular)?

Changing the current --metrics/-m to behave like -M would be ideal, but breaks backward compatibility.

[jorgeorpinel]

One last comment for now, sorry:

Implement continuous metrics with dvc viz command.

Keep in mind some of the text deleted in #1097 could be restored when writing docs for dvc viz (or whatever the command name ends up being). E.g. https://github.com/iterative/dvc.org/pull/1097/files#diff-5941ad4a74935aeef222249cb407d385L72-L74