SMAC for cheap evaluation functions

[jendrikseipp]

Description

I'm using SMAC4AC for a scenario with 180 hyperparameters and a cheap evaluation function (no instances). The evaluation function takes ~0.5 seconds on average. I tried SMAC4HPO first, but it can only handle at most 40 parameters. Out of the 36000 seconds overall wall-clock limit, SMAC uses only 220 seconds for 530 evaluation runs. Do you have a recommendation on how to tweak SMAC for this scenario, i.e., speeding up the selection of configurations to test, for example, by sacrificing prediction quality?

I have already tried to adapt the value for acq_opt_challengers, but it didn't speed things up, even in a test scenario with 36 parameters and an empty evaluation function:

acq_opt_challengers	function evaluations
100K	41
10K	63
1K	67
10	65
1	57

Versions

SMAC 0.12.0 ConfigSpace 0.4.12

[mlindauer]

I are aware that SMAC has some performance bottlenecks for very cheap functions, but I'm surprised that it could be so bad. Do you optimize solution quality or runtime? If you optimize for runtime, could you please also give it a try with the current dev branch? I added some performance improvements which will be part of the next release.

In addition, do you have a minimal working example for reproducing it?

Best, Marius

[jendrikseipp]

Thanks for the quick answer, Marius! I'm optimizing for solution quality. Here is a minimal working example. Called with ./mwe.py --random-seed=0 --modules=20 --overall-time-limit 30, it uses 20*9 hyperparameters and checks one configuration in 30 seconds

#! /usr/bin/env python3

"""
Optimize robot configuration with SMAC.
"""

import argparse
import logging
import math
import os.path
import sys
import warnings

warnings.simplefilter(action="ignore", category=FutureWarning)

import numpy as np

from smac.configspace import ConfigurationSpace
from smac.scenario.scenario import Scenario
from smac.facade.smac_ac_facade import SMAC4AC
from smac.initial_design.random_configuration_design import RandomConfigurations

from ConfigSpace.hyperparameters import CategoricalHyperparameter
from ConfigSpace.hyperparameters import UniformFloatHyperparameter

DIR = os.path.abspath(os.path.dirname(__file__))


def parse_args():
    parser = argparse.ArgumentParser(description=__doc__)

    parser.add_argument(
        "--modules",
        type=int,
        default=10,
        help="Number of modules (default: %(default)s)",
    )

    parser.add_argument(
        "--evaluations",
        type=int,
        default=sys.maxsize,
        help="Number of configurations to evaluate (default: %(default)s)",
    )

    parser.add_argument(
        "--overall-time-limit",
        type=float,
        default=20 * 60 * 60,
        help="Maximum total optimization time (default: %(default)ss)",
    )

    parser.add_argument("--debug", action="store_true", help="Print debug info")

    parser.add_argument(
        "--random-seed",
        type=int,
        default=0,
        help="Initial random seed for SMAC and our internal random seeds (default: %(default)d)",
    )

    parser.add_argument(
        "--smac_output_dir",
        default="smac",
        help="Directory where to store logs and temporary files (default: %(default)s)",
    )

    parser.add_argument(
        "--headless", default=False, action="store_true", help="Don't show animation"
    )
    return parser.parse_args()


ARGS = parse_args()


def setup_logging():
    """
    Print DEBUG and INFO messages to stdout and higher levels to stderr.
    """
    # Python adds a default handler if some log is generated before here.
    # Remove all handlers that have been added automatically.
    logger = logging.getLogger("")
    for handler in logger.handlers:
        logger.removeHandler(handler)

    class InfoFilter(logging.Filter):
        def filter(self, rec):
            return rec.levelno in (logging.DEBUG, logging.INFO)

    logger.setLevel(logging.DEBUG if ARGS.debug else logging.INFO)

    formatter = logging.Formatter("%(asctime)s %(levelname)-8s %(message)s")

    h1 = logging.StreamHandler(sys.stdout)
    h1.setLevel(logging.DEBUG)
    h1.addFilter(InfoFilter())
    h1.setFormatter(formatter)

    h2 = logging.StreamHandler()
    h2.setLevel(logging.WARNING)
    h2.setFormatter(formatter)

    logger.addHandler(h1)
    logger.addHandler(h2)


setup_logging()

# SMAC moves old directories out of the way, but we want a completely pristine directory to safeguard against errors.
if os.path.exists(ARGS.smac_output_dir):
    sys.exit("Error: SMAC output directory already exists")


def evaluate_cfg(cfg):
    return 7


POSITIONS = ["left", "right", "top"]


def build_configuration_space():
    # Build configuration space which defines all parameters and their ranges.
    cs = ConfigurationSpace()

    for module_index in range(ARGS.modules):
        # Production rule
        module_list = list(range(ARGS.modules)) + ["none"]
        for direction in POSITIONS:
            param = CategoricalHyperparameter(
                f"{direction}-{module_index}", choices=module_list
            )
            cs.add_hyperparameter(param)

        def add_float(name, lower, upper):
            cs.add_hyperparameter(
                UniformFloatHyperparameter(
                    f"{name}-{module_index}", lower=lower, upper=upper
                )
            )

        # Controller
        add_float("amp", lower=0, upper=1)
        add_float("phase", lower=-1, upper=1)
        add_float("offset", lower=-math.pi, upper=math.pi)
        add_float("frequency", lower=0.005, upper=0.1)

        # Module parameters
        add_float("angle", lower=-math.pi / 2, upper=math.pi / 2)
        add_float("size", lower=0.25, upper=1)

    return cs


config_space = build_configuration_space()

print("Number of hyperparameters: ", len(config_space.get_hyperparameters()))

scenario = Scenario(
    {
        "run_obj": "quality",
        # maximum number of function evaluations
        "ta_run_limit": ARGS.evaluations,
        # Make sure that we stop eventually if we don't hit the evaluations limit.
        "wallclock_limit": ARGS.overall_time_limit,
        "cs": config_space,
        "deterministic": "true",
        # memory limit for evaluate_cfg
        "memory_limit": None,
        # time limit for evaluate_cfg
        "cutoff": None,
        "output_dir": ARGS.smac_output_dir,
        # Disable pynisher.
        "limit_resources": False,
    }
)

smac = SMAC4AC(
    scenario=scenario,
    initial_design=RandomConfigurations,
    rng=np.random.RandomState(ARGS.random_seed),
    tae_runner=evaluate_cfg,
)
print("Optimizing...")
incumbent = smac.optimize()

print("Final configuration: {}".format(incumbent.get_dictionary()))

[jendrikseipp]

It seems almost all of the time is spent in LocalSearch._do_search(). Do you have some tips on how to spend less time there?

[mlindauer]

I think I found the problem. Could you please add one line nearly at the end of your code:

    scenario=scenario,
    initial_design=RandomConfigurations,
    rng=np.random.RandomState(ARGS.random_seed),
    tae_runner=evaluate_cfg,
)

smac.solver.scenario.intensification_percentage = 0.5 # <-------

print("Optimizing...")
incumbent = smac.optimize()

Since you have a deterministic target function and you don't optimize (and care) for runtime, SMAC assumes that your target function is fairly expensive (otherwise you want to look for other optimizers) compared to SMAC's overhead. In this case we disable the balance between overhead and number of target function evaluations. By adding this line, SMAC should roughly use again the same amount of time for its internal optimization and the target function calls.

Best, Marius

[jendrikseipp]

Thanks! For the 30 second test run with an empty evaluation function this version still only tries 3 configurations, but I guess the hope is that SMAC balances its overhead and the function evaluations over longer time periods. I'll try it for a 1h time limit.

[jendrikseipp]

The results are in. The time for running function evaluations remains roughly the same as before, but almost twice as many configurations are evaluated in the same time. So it seems SMAC prefers cheaper-to-evaluate configurations now. Was that the intended effect?

I'm running an experiment with different values for intensification_percentage. Other than this parameter, is there maybe a more direct way of reducing the time spent for the local searches?

[mfeurer]

I'm afraid that intensification_percentage is broken for the current master. You would need to install the development branch from github.

[jendrikseipp]

Thanks for the hint! I'll try the development branch.

[jendrikseipp]

The intensification_percentage seems to work in the development branch. Now I can nicely control SMAC's overhead. Thanks for your help, guys!

[jendrikseipp]

I tried using pSMAC for the same optimization. However, even when using the recent revision d091bd35d8b971ee97039a07d3e4bd2f605a7015 from the development branch and explicitly setting intensification_percentage to 0.5, the time for function evaluations only amounts to ~3% of the wall-clock time. The only change I made is adding "shared_model": True, "input_psmac_dirs": <output_path> to the Scenario options and assigning the SMAC runs IDs. Do you know how to fix this?

[jendrikseipp]

Could it be that the same fix for intensification_percentage that was made in the development branch has to be applied to pSMAC as well?

[benjamc]

We don't support pSMAC in SMAC3-2.0 anymore, however we now support parallelism natively! For the case of having a large configuration space with cheap evaluation functions, you can set the argument retrain_after in the config selector . In this case, set it high such that the surrogate model is not trained too often.