Default parameter value is ignored for initial design

[jendrikseipp]

Steps/Code to Reproduce

import logging

import numpy as np

from ConfigSpace.hyperparameters import UniformIntegerHyperparameter

from smac.configspace import ConfigurationSpace
from smac.scenario.scenario import Scenario
from smac.facade.smac_hpo_facade import SMAC4HPO


def evaluate_cfg(cfg):
    print(cfg)
    return None

logging.basicConfig(level=logging.INFO)

cs = ConfigurationSpace()

cs.add_hyperparameters([
    UniformIntegerHyperparameter("a", lower=0, upper=600, default_value=1),
])

scenario = Scenario({
    "run_obj": "quality",
    # max. number of function evaluations
    "runcount-limit": "inf",
    "cs": cs,
    "deterministic": "true",
    # memory limit for target algorithm (3.5 GB)
    "memory-limit": 3584,
})

default_cfg = cs.get_default_configuration()
print("Default config:", default_cfg)

print("Optimizing...")
smac = SMAC4HPO(
    scenario=scenario, rng=np.random.RandomState(42), tae_runner=evaluate_cfg)
incumbent = smac.optimize()
inc_value = evaluate_cfg(incumbent)
print("Optimized value: %.2f" % (inc_value))

Expected Results

The default value 1 is used for the initial design.

Actual Results

INFO:smac.utils.io.cmd_reader.CMDReader:Output to smac3-output_2019-09-13_08:45:09_553534
Default config: Configuration:
  a, Value: 1

Optimizing...
INFO:smac.facade.smac_hpo_facade.SMAC4HPO:Optimizing a deterministic scenario for quality without a tuner timeout - will make SMAC deterministic and only evaluate one configuration per iteration!
INFO:smac.initial_design.sobol_design.SobolDesign:Running initial design for 10 configurations
INFO:smac.facade.smac_hpo_facade.SMAC4HPO:<class 'smac.facade.smac_hpo_facade.SMAC4HPO'>
Configuration:
  a, Value: 300

For some reason the default value 1 is ignored and instead (lower+upper)/2 is used as the default value.

Versions

0.11.0

[mlindauer]

Dear Jendrik,

Thank you for reporting the issue. I can confirm that the default configuration is ignored if the SMAC4AC facade is not used or if the initial design is not explicitly set to the default initial design (in case of the SMAC4HPO or SMAC4BO facade).

I have to discuss this with the other developers, because it will change the results on some standard benchmarks quite a bit; but my intuition is that we want to change that in the next release.

Best, Marius

[jendrikseipp]

Thanks for the info!

[stheid]

Is there any workaround? my searchspace is very large. i would like to give smac at least a little hint by starting close to a usefull spot.

[mfeurer]

Not yet, but a pull request would be very welcome (it should add this as an option, though).

[stheid]

Workaround A

Modified sobol_design.py by prepending the default configuration https://github.com/automl/SMAC3/blob/2b27e0d063bdb485e07a7b76e9ce41e05cfbbf71/smac/initial_design/default_configuration_design.py#L27-L29

Workaround B

replace 'SobolDesign' with 'None' https://github.com/automl/SMAC3/blob/2b27e0d063bdb485e07a7b76e9ce41e05cfbbf71/smac/facade/smac_hpo_facade.py#L39

Cause

I tracked down the bug. however, i am not able to predict the intended behavior.

Both Bayesian optimizers select Sobol as initial design: https://github.com/automl/SMAC3/blob/2b27e0d063bdb485e07a7b76e9ce41e05cfbbf71/smac/facade/smac_hpo_facade.py#L39 https://github.com/automl/SMAC3/blob/2b27e0d063bdb485e07a7b76e9ce41e05cfbbf71/smac/facade/smac_bo_facade.py#L56

Therefore the initial_design is not None and the value for initial_incumbant in scenario is ignored https://github.com/automl/SMAC3/blob/2b27e0d063bdb485e07a7b76e9ce41e05cfbbf71/smac/facade/smac_ac_facade.py#L427-L430

additionally Sobol design https://github.com/automl/SMAC3/blob/2b27e0d063bdb485e07a7b76e9ce41e05cfbbf71/smac/initial_design/sobol_design.py#L46-L50 does not use the default configuration.

I hope that helps others in tackle the issue.

[stheid]

The documentation is not very verbose on what Sobol even is.

[jendrikseipp]

I used the following workaround:

from smac.facade.smac_hpo_facade import SMAC4HPO
from smac.initial_design.default_configuration_design import DefaultConfiguration
smac = SMAC4HPO(
    initial_design=DefaultConfiguration,
   ...
)

[stheid]

Has basically the same effect as Workaround B, but yours is obvoiusly better.

[stheid]

@mfeurer is it recommended to disable sobol like https://github.com/automl/SMAC3/issues/533#issuecomment-553547011

[mlindauer]

It depends on what you want to achieve.

The HPO-facade uses a sobol sequence but in turn, it also uses less interleaved random configurations. Thus, if you disable the sobol sequence (workaround b) and you still use the HPO facade, SMAC will be more greedy compared to its default settings.

Workaround A is most likely what you want in most cases.

[stheid]

Thanks for the information.

workaround A leads SMAC to prematurely stop after it finished its n+1 initial configurations (while n is the number SOBOL calcualtes from the time budget)

so for 100 runs, it will stop after 25+1=26

[mfeurer]

@gamer01 could you please open a new issue on SMAC stopping prematurely with code to reproduce this?

[stheid]

It only happens for my modified version. Therefore i would not deem it a bug in smac. (The Initializer seems to assume it has 25 elements, as it has 26 now, it stops)

[stheid]

solved by skiping one sobol element

MySobol.py.txt

from MySobol import SobolDesign
smac = SMAC4HPO(scenario=scenario, tae_runner=tae, initial_design=SobolDesign)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[timruhkopf]

Good things may take a while, but with our latest release (2.0), this becomes an easy-to-fix issue:

simply add to the <facade>.get_initial_design call (e.g. in this SVM example) the default config as an additional_configs like this:

classifier = SVM()

[....]

 initial_design = HyperparameterOptimizationFacade.get_initial_design(
        scenario,
        n_configs=5,
        additional_configs=[  # to get the default configuration in the initial design
            classifier.configspace.get_default_configuration()
        ]
    )

This will result in SMAC executing the n_configs from the initial design and, subsequently, the default config you specified.

Notice, that it is still not default behaviour to execute the configspace default (if available)

[timruhkopf]

At long last, this is now an optional feature; see PR #995.

Setting Scenario([...], use_default_config=True) will make use of the default config. but it is not the default for legacy reasons, particularly because of the benchmarks, which would be solved by the default config as per Marius' comment above:

Dear Jendrik,

Thank you for reporting the issue. I can confirm that the default configuration is ignored if the SMAC4AC facade is not used or if the initial design is not explicitly set to the default initial design (in case of the SMAC4HPO or SMAC4BO facade).

I have to discuss this with the other developers, because it will change the results on some standard benchmarks quite a bit; but my intuition is that we want to change that in the next release.

Best, Marius