[GENERAL SUPPORT]: Recommendations for "in-situ" changes to optimization configurations in closed-loop environments

[sgbaird]

Question

I imagine Meta AE team has run into this situation in practice at various points:

• a closed-loop optimization campaign is running (for clarity, we'll say it's a hyperparameter tuning problem)
• it is decided that hyperparameter X should also be included in the search space
• now, there's a mismatch between how the prior data and the current data is represented

Please provide any relevant code snippet if applicable.

Code of Conduct

• [x] I agree to follow this Ax's Code of Conduct

[jakobzeitler]

This is AFAIK a fundamental problem for any method of experimentation, not just Bayesian Optimisation.

Unless you can somehow recover, estimate or guesstimate the values of the new hyperparamter X for each historical sample, you are strictly speaking looking at a different experimental setup, and as such are starting a new campaign with 0 samples.

In the world of Bayesian Optimisation, the closest work I am aware of to tackle this challenges is available here: https://openreview.net/forum?id=FSqIx6FO0O

[sgbaird]

Nice, but just noting that I'm talking from a plumbing / architecture perspective, not a modeling perspective.

[Balandat]

Yes, this is a pretty common occurrence in practice. As @jakobzeitler indicates above, we do consider this a problem of transfer learning with heterogeneous search spaces. We do have models and logic for this but I don't think it's fully available in OSS - @saitcakmak can you confirm?

[saitcakmak]

The BoTorch models for heterogeneous search space TL is available in this repo (https://github.com/saitcakmak/heterogeneous_botl) but the Ax integration for transfer learning methods is not available in OSS.

[sgbaird]

Right, though what about from a plumbing/API perspective, nothing model-based, and assuming it's just a different representation of the search space, etc. For example, someone wanting to change bounds on a hyperparameter or including an ingredient into the search space that they originally didn't include (i.e., zero for the rest).