[bug] Small ConfigSpace with large runcount-limit does not terminate.

[Strandtasche]

Description

When optimizing a ConfigSpace with fewer elements within than the scenario's runcount-limit the optimization does not terminate. We believe the problem might be that after exhausting every possible Config in the ConfigSpace it's trying to sample a new Configuration but since there are none it is stuck in an endless loop.

Steps/Code to Reproduce

Taking the SVM example from the repository. Reduce the size of the ConfigSpace by removing Line 71-92. Now there are only 4 possible configurations. When setting the runcount-limit in Line 96 to 4 or less, everything works as intended, but setting it to 5 or more leads to the program not terminating anymore.

As a further evidence for or theory what goes wrong: when changing the size of the Configspace by adding or removing categories the threshold whether or not the program terminates changes in accordance with our guess.

For ease or reproduction here is a gist of the example with the aforementioned changes.

Expected Results

One would expect the optimization to terminate when the ConfigSpace is empty.

Actual Results

The optimization never terminates and is permanently stuck

Versions

0.13.1

SWIG was installed using conda instead of aptitude

[mlindauer]

Hi,

Thanks for reporting this issue. In fact, SMAC does not track whether the configuration space was exhaustively covered, and thus SMAC will not terminate in this case. Fixing this issue for the general case is unfortunately not easy and therefore, we have not addressed this issue so far.

If you are aware that your configuration space space consists of less configuration than your configuration budget, we recommend that you fix the way you call SMAC.

Best, Marius

[mfeurer]

Could the two of you please check if this is the same as #21?

[Strandtasche]

I think @KEggensperger's problem might be the same one I encountered. Somehow I didn't find their ticket when I searched for existing Issues.

[mfeurer]

In case anyone in interested, there will soon be a function to create the full grid of legal configurations via the configuration space package https://github.com/automl/ConfigSpace/pull/142 and this could be used to fix this bug.

[sarah-segel]

Will be addressed in #959, thus this issue is closed.