Noise and down-sampling examples for objective function

[JohnReid]

I have an objective that depends on many test examples (an AUC-like score). I can evaluate the objective function on any number of examples but the noise is greater when there are fewer examples. Is there anyway spearmint can take advantage of this? Perhaps it can/could run the objective on a few examples when it is unsure about large parts of parameter space and run on more examples when it has a better idea where the optimal parameters are? Here the number of examples isn't a parameter spearmint wants to optimise, rather it is a parameter that relates linearly to the expected running time and inversely to the noise level. Not sure how easy it would be to implement but it could be useful.

[JasperSnoek]

Interesting.... Do you mean by 'examples' the number of evaluations that spearmint has observed or some quantity that pertains to your code that spearmint is calling? Certainly the noise will be greater when spearmint observes fewer examples. Ideally spearmint should already be taking this into account - when there is more uncertainty over the space it will explore more.

On Fri, Oct 18, 2013 at 9:53 AM, JohnReid [email protected] wrote:

I have an objective that depends on many test examples (an AUC-like score). I can evaluate the objective function on any number of examples but the noise is greater when there are fewer examples. Is there anyway spearmint can take advantage of this? Perhaps it can/could run the objective on a few examples when it is unsure about large parts of parameter space and run on more examples when it has a better idea where the optimal parameters are? Here the number of examples isn't a parameter spearmint wants to optimise, rather it is a parameter that relates linearly to the expected running time and inversely to the noise level. Not sure how easy it would be to implement but it could be useful.

— Reply to this email directly or view it on GitHubhttps://github.com/JasperSnoek/spearmint/issues/13 .

[JohnReid]

By examples I mean test cases or something that pertains to my code. My situation is analogous to the following: suppose I have a hand-written digit classifier and a million hand-written digits. Now I might not want to evaluate the classifier on all one million digits every time spearmint calls the evaluation function but as spearmint refines its estimate of the best parameters I might want to use more and more digits (examples/test cases) to reduce the noise associated with each evaluation and allow spearmint to accurately find the best parameters. I understand there is a trade-off there but it seems that as spearmint explicitly models the noise, it might be in a good position to take advantage of this in some way.