Evaluating the default configuration in BOHB

[ashraaghav]

I tried using CAVE with BOHB and I noticed that the default configuration is not used in the report at all, unlike in SMAC3. I understand it is because BOHB does not necessarily evaluate the default configuration, but it will be nice to compare the incumbent performance's with the default's in CAVE, especially in the configurator footprint plot. Is it possible in CAVE?

[mlindauer]

Hi,

Thank you for your feedback. Unfortunately, so far it is not possible because CAVE does not evaluates any configuration but it only uses the data provided to it. We could use a performance estimate from an EPM; but since BOHB does not use the default configuration anyway, there is no point in comparing the default against BOHB's incumbent. Maybe if we could force BOHB (in the future) to consider a given default configuration, we could do a similar treatment like we do with SMAC3.

Overall, I see where you are coming from. We will discuss this internally again, but it is unlikely that we will provide a solution in the near future.

Best, Marius