[FEATURE REQUEST]: FixedFeatureModel

[jduerholt]

Motivation

I saw the paper on sustainable concrete, and I really like it. I especially like the FixedFeatureModel as it allows for a very flexible setup of the optimization problem. Do you plan to include the FixedFeatureModel directly into BoTorch? @SebastianAment

Describe the solution you'd like to see implemented in BoTorch.

FixedFeatureModel directly implemented in BoTorch.

Describe any alternatives you've considered to the above solution.

No response

Is this related to an existing issue in BoTorch or another repository? If so please include links to those Issues here.

No response

Pull Request

Yes

Code of Conduct

• [x] I agree to follow BoTorch's Code of Conduct

[jduerholt]

Another option would be to use the AppendFeatures input transform ..., but I am not sure what should be the recommended way for doing this. Here one would definitely run into problems with the indices, using the current implementation. So maybe, this kind of wrapping using the FixedFeatureModel would make it very easy to setup optimizations like this.

[Balandat]

At this point we don't have any plans to include it, but I'd be willing to do so if works generically across all (or at least most) use cases.

I do like the idea of enabling this via an input transform. How hard would it be to update AppendFeatures to behave similarly to the FixedFeatureModel without running into indexing issues? We could call this InsertFeatureTransform to indicate that it's not strictly appending.

[Balandat]

For reference, FixedFeatureModel is defined here: https://github.com/facebookresearch/SustainableConcrete/blob/ed62256b9e0d126c346cb705170e595891923e6d/models.py#L136

[SebastianAment]

I do like the idea of enabling this via an input transform. How hard would it be to update AppendFeatures to behave similarly to the FixedFeatureModel without running into indexing issues? We could call this InsertFeatureTransform to indicate that it's not strictly appending.

I think we could encapsulate it in a fixed features transform, and this in turn could be used to get a fixed features model, and unify the implementation with the fixed features acquisition function. Pushing this down to an input transform makes the most sense to me and enables more flexibility.

Can definitely look at incorporating this into BoTorch proper, we might have some other internal applications for this coming up.

@jduerholt, do you have current applications for this type of model as well?

[jduerholt]

@Balandat

I do like the idea of enabling this via an input transform. How hard would it be to update AppendFeatures to behave similarly to the FixedFeatureModel without running into indexing issues? We could call this InsertFeatureTransform to indicate that it's not strictly appending.

I can have a look!

@SebastianAment

Can definitely look at incorporating this into BoTorch proper, we might have some other internal applications for this coming up.

This would be of course great. Maybe we should give a short indication to each other, when someone of us is really starting to do this to prevent double work ;)

you have current applications for this type of model as well?

I am indeed working currently on a (mathematically) similar problem/project and had some ideas in the same direction as you in the paper, but was not able to explore them due to hard time constraints in the project and very messy historical data, so I was spending most of the time in tidying up data :D In addition to some other problems, it is a higher dimensional problem and I see the best performance with the SAAS model (side node to @Balandat: it would be really cool to allow for more flexible kernel design in fully bayesian models, despite knowing that it is kind of komplex due to pyro under the hood). Despite of this project, I see broad applicability in a lot of reaction optimization use cases when you explicitly want to model the time. Btw: have you thought on applying this kind of kernel to learning curves?

One question to the paper: how much better is your model vs two independent models? One for 1 day? And one for 28 days? How much improvement to you get given the additional complexity of the models and the optimization?