biome-text icon indicating copy to clipboard operation
biome-text copied to clipboard

Invalid metric warning in HPO

Open ignacioct opened this issue 3 years ago • 1 comments

Is your feature request related to a problem? Please describe. When specifying a metric in an HPO run with ray tune, we could try to check if that metric exists and can be used. Maybe is out of our hand, as this is passed to tune.run(), but biome is running in the background so maybe that metric can be checcked

Describe the solution you'd like Imagine:

analysis_frozen = tune.run( tune_exp, scheduler=tune.schedulers.ASHAScheduler(), metric="totally_fake_metric", mode="max", progress_reporter=tune.JupyterNotebookReporter(overwrite=True), )

We could print a warning/error.

ignacioct avatar Mar 27 '21 20:03 ignacioct

This is just a reminder for me: The head should store the logged metric names + the information if shown in the pbar or not. This would make a quick check of the specified monitored metric very easy.

dcfidalgo avatar Apr 21 '21 14:04 dcfidalgo