dvc
dvc copied to clipboard
dvc experiments checkout: option to select the best experiment regarding one metric
I had an idea while trying the new experiment feature. For the use case of grid-search or hyper-parameter optimization, could we imagine an option for dvc experiments checkout to checkout the best experiment regarding one of the metrics? Something like dvc experiments checkout --best metrics.json:accuracy. With an option like this, you can run (manually or automatically) a lot of experiments and ask dvc to return the best experiment, which can be then commited to git. Or instead of --best, it could be --lowest or --highest to take into account that some metrics should be minimized and some others maximized.
Commenting on a workaround for reference, in case someone reaches this issue.
Behavior like the description can be achieved (in a UNIX system) as follows:
dvc exp apply $(dvc exp show --csv --sort-by metrics.json:accuracy | tail -n 1 | cut -d , -f 1)
Related: #8775.