Skip to content

dvc experiments checkout: option to select the best experiment regarding one metric #4283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
gcoter opened this issue Jul 25, 2020 · 2 comments
Labels
A: experiments Related to dvc exp p3-nice-to-have It should be done this or next sprint ui user interface / interaction

Comments

@gcoter
Copy link
Contributor

gcoter commented Jul 25, 2020

I had an idea while trying the new experiment feature. For the use case of grid-search or hyper-parameter optimization, could we imagine an option for dvc experiments checkout to checkout the best experiment regarding one of the metrics? Something like dvc experiments checkout --best metrics.json:accuracy. With an option like this, you can run (manually or automatically) a lot of experiments and ask dvc to return the best experiment, which can be then commited to git. Or instead of --best, it could be --lowest or --highest to take into account that some metrics should be minimized and some others maximized.

@ghost ghost added the triage Needs to be triaged label Jul 25, 2020
@pmrowla pmrowla added enhancement Enhances DVC A: experiments Related to dvc exp p2-medium Medium priority, should be done, but less important ui user interface / interaction labels Jul 27, 2020
@ghost ghost removed the triage Needs to be triaged label Jul 27, 2020
@pmrowla pmrowla added p3-nice-to-have It should be done this or next sprint and removed p2-medium Medium priority, should be done, but less important labels Aug 18, 2020
@daavoo
Copy link
Contributor

daavoo commented Apr 17, 2023

Commenting on a workaround for reference, in case someone reaches this issue.

Behavior like the description can be achieved (in a UNIX system) as follows:

dvc exp apply $(dvc exp show --csv --sort-by metrics.json:accuracy | tail -n 1 | cut -d , -f 1)

@dberenbaum
Copy link
Contributor

Related: #8775.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: experiments Related to dvc exp p3-nice-to-have It should be done this or next sprint ui user interface / interaction
Projects
None yet
Development

No branches or pull requests

5 participants