Hyperopt analysis

[Cmurilochem]

I am starting the implementation of @juanrojochacon's hyperopt algorithm wherein data of 1/$\varphi^{2}$ is used to decide on the best $\chi^{2}$ hyperpoint.

I realized that all data needed are contained within tries.json as implemented here in hyperopt_loss branch.

So, I was wondering that the best approach would be to implement this post-hyperopt analysis in an external script like those in validphys. I see here some options:

• it could be a post-hyperopt.py similar to postfit.py
• an option in vp_hyperoptplot.py and implemented in hyperoptplot.py
• inside hyper_algorithm.py (my first option?)

Which one would you suggest ? Any other ideas are also very welcome.

Edited: The major advantage I see in having such external script is that we would be to recover any possible statistics we want with just one hyperopt experiment.

[scarlehoff]

Please, when computing the 1/phi^2, compute separately the phi^2 of the data that went into training and the phi^2 of the folds as well, so we can also test afterwards the phi^2 stability.

This kind of statistics I think it is better suited as part of vp_hyperoptplot. That script already allows you to do a post-hyperopt selection similar to postfit (like selecting whether to average the loss or select the best-worst, etc) so it would be just another option there.

[Cmurilochem]

Thanks @scarlehoff. It sounds good. I am currently just evaluating phi^2 in the held out fold (hyper_losses_phi2). But I will include these extra phi's as suggested and proceed to vp-hyperoptplot.py.

[RoyStegeman]

Eventually it indeed makes sense for it to become part of vp-hyperoptplot, there we already support various options for metrics with some way to choose between them: average, min and max or the validation fraction. But since at the moment we're just trying this metric as one possibility but may decide to use something else in end, I'd suggest just doing whatever makes your life easier for now