facebook
[Question] Best practices regarding Sobol sampling
In the gpei_hartmann_service.ipynb tutorial, before exploring using GPEI, a Sobol strategy has been used. Just curious if there is any reference on the best practices regarding the Sobol strategy: mainly what's the relationship between the search space and the number of Sobol trials.
In general, you should get away with relatively few Sobol points. This is really just an initialization and mostly in place to prime the model to be able to fit to some data. In general, if there are too few points to get a good model fit, the model uncertainty should be high across the entire search space and cause any reasonable acquisition function to behave more or less like random exploration. So if you have a small iteration budget it would be reasonable to use a small number of Sobol points. Using a few more points will probably reduce the chances of the model running into numerical issues and make the model fitting more stable (this is more an intuition / hypothesis than proven fact though, I've never formally investigated this).
Interestingly, for GPEI there were nearly identical results using a 10/90 Sobol/GPEI vs. a 46/54 Sobol/GPEI with 23 hyperparameters, see Figure 2 of https://doi.org/10.1016/j.commatsci.2022.111505.
@tramchamploo Sobol is supported and BayesOpt has some support for hierarchical search spaces. See https://github.com/facebook/Ax/issues/140 (related, but closed in favor of #140 is https://github.com/facebook/Ax/issues/327).
