Exhausted ConfigSpace

[mlindauer]

If SMAC3 tried all possible configurations in the ConfigSpace (and the target algorithm is deterministic), but the configuration budget is not yet exhausted (e.g., number of function evaluations), SMAC3 will never terminate.

Having an approximation of the configspace size would help to terminate SMAC, if the configspace is exhausted.

Related to this issue, even if the configuration budget is appropriated, it can be hard for SMAC to sample a new configuration which it has not yet run. The SMAC2 solution for this case is to count the number of calls of intensify without adding a new algorithm run in the runhistory. Do we want to also include this check in SMAC3?

I added the example leadingones.py to reproduce these issues.

[mlindauer]

So, how to solve this problem? I don't know whether we already agreed on something.

I would propose to add a check in Intensifier that if intensifiy was called 10 times in a row without any new algorithm runs, we will terminate SMAC with a message like

Unable to sample new configurations - Exit

What do you think?

[mfeurer]

without any new algorithm runs

do you mean intensify was called and didn't add a new run to the runhistory?

If the configuration space is finite, we could add some way of enumerating them, and then generating them by their index. This maybe makes sense for spaces up to 10.000 configurations. After that, we maybe want to traverse all possible configurations but only keep a certain number (a total of 10.000).

A related problem: What should we actually do if the space is not finite, but the only continuous parameter is a conditional which is sampled only rarely?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alexandertornede]

#718

[sarah-segel]

Will be addressed in #959, thus this issue is closed.