[Feature Request]Fantasize for HeteroskedasticSingleTaskGP

[jinjiaodawang]

🚀 Feature Request

If KG is applied to a heteroskedastic model, it will raise a NotImplementedError that comes from the following function. code

Is there any plan to implement this part? If I write the a fantasize function for hetero-model like the author did in FixedNoiseGP, will it work? Thank you.

[eytan]

IIRC @danielrjiang and @Balandat had done some experiments w/ KG + hetGP models using a GP to model the variance. They might be able to give some advice sometime after the holiday.

[Balandat]

We don't currently have concrete plans for this. The way HeteroskedasticSingleTaskGP models the variance internally makes it quite hard to do this "properly" - essentially conditioning on observations would require not just fantasizing on the overall model itself, but also fantasizing on the noise model. While that's probably doable, I'm a bit concerned that this will end up being really slow. As a first order approximation one could probably just try leaving the noise model untouched and only fantasize on the outer model.

Is there a specific reason you want to use KG with the HeteroskedasticSingleTaskGP? If you can live with using something like EI that doesn't require fantasizing, then things would be a lot easier.

[jinjiaodawang]

My current work is an academic one , which involves comparing the performance of these acquisition functions. I actually used apporoximation like "leaving the noise model untouched and only fantasize on the outer model", the performance is not quite ideal so I decided to raise this request. I will think of how exact fantasize works. It would be better if this feature is officially implemented. Thank you for your response.

[trbedwards]

I would also very much like this feature. I'm attempting to build a Bayesian Optimisation routine for 'expensive' simulations that take on the order hours to run; I'd like to run multiple simulations at a time (maybe 10-20 or so) where there are typically 10-20 parameters to optimise for.

I'm not using knowledge gradients but instead I'm using EI with a "Kriging Believer" heuristic for asynchronous batch fake observations (this means conditioning the GP with the posterior mean until the simulations for the current batch finish, and then undo that conditioning by pulling out an old copy of the GP and subsequently conditioning that with the 'real' observations from the simulations). I find this works much better than using qEI (it optimises under less number of iterations) and has the advantage of being asynchronous so I can maximise load balance.

Currently I'm using the default homoskedastic noise model, but that leads to a breakdown in the GP prediction performance when a stray simulation observation goes way out of the GP uncertainty envelope and beyond the noise error bars. This forces either the GP length scales to decrease or the homoskedastic noise level to increase, leading a breakdown in predictive ability. I think the solution to this will be to be able to model heteroskedastic noise to account for parts of the parameter space which are particularly noisy. However, I'm unable to do that with the "Kriging Believer" heuristic because it requires the condition_on_observations method to be implemented.

For my application I'd like to implement a workaround by simply copying the condition_on_observations method from SingleTaskGP, but then apply the same thing to gp.likelihood.noise_covar. I'd like to ask the same question the original author of this issue asked again; "Will it work"?. I'm not worried about performance since I'm using EI rather than something heavyweight like KG, and my function evaluations (simulations) take a long time to execute anyway.