SMAC4* cannot handle functools.partial functions and passes undocumented **kwargs to the target-function (lambdas do work)

[stheid]

Description

https://github.com/automl/SMAC3/blob/2b27e0d063bdb485e07a7b76e9ce41e05cfbbf71/examples/SMAC4HPO_svm.py#L28 and https://github.com/automl/SMAC3/blob/2b27e0d063bdb485e07a7b76e9ce41e05cfbbf71/examples/SMAC4HPO_svm.py#L105 suggests, that the target callable only needs to accept one argument.

However #546 shows, that this is not the case for SMAC4BO

Steps/Code to Reproduce

see #546

Expected Results

target-function is evaluated by passing only the configuration as argument

Actual Results

fails because the arguments seed and instance are not accepted by the target-function.

Workarround

add **kwargs argument to target-function. see refered issue for details

Versions

0.11.1

[mlindauer]

Hi,

Thank you for reporting this issue. However, I cannot reproduce it on my end. Running one of the examples, e.g., the svm example (https://github.com/automl/SMAC3/blob/master/examples/SMAC4HPO_svm.py) works fine on my machine. The target function in this example, indeed only expects one argument.

Best, Marius

[stheid]

The issue is not related to HPO but to SMAC4BO! as i write in the other issue, i cannot run HPO because of the swig issue on my machine. However it also does not happen when i simpy change the optimizer in the example you linked. I will try to hunt it down further and inform you if can create a simpler case to produce the issue.

[mlindauer]

the TAE module is independent of the used SMAC mode (AC, HPO, BO). Therefore, it should work fine with all or with none. I also verified that I can run the rosenbrock example (again only one argument for the target function): https://github.com/automl/SMAC3/blob/master/examples/SMAC4BO_rosenbrock.py

[stheid]

The issue is related to my partial function!

change target function signature

def svm_from_cfg_moreargs(cfg, x, y):

set the default arguments using functools.partial

svm_from_cfg = partial(svm_from_cfg_moreargs, x=1, y=2)

for some reason this results in the even more arguments for the function under the hood

    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/site-packages/pynisher/limit_function_call.py", line 93, in subprocess_func
    return_value = ((func(*args, **kwargs), 0))
TypeError: svm_from_cfg_moreargs() got an unexpected keyword argument 'seed'

Allowing more kwargs to be added solves the issue:

def svm_from_cfg_moreargs(cfg, x, y,**kwargs):

[mlindauer]

If I remember correctly, instead of using partial using lambda worked fine.

[mfeurer]

I think the underlying issue here is that we assume that if the target function has more than one argument that these are either instance or seed, and that we check that via inspection. We actually changed this in the development branch (to only pass these if the argument has the correct name), but we should document this somewhere.

Current code (not on PYPI): https://github.com/automl/SMAC3/blob/development/smac/tae/execute_func.py#L74

Code on pypi: https://github.com/automl/SMAC3/blob/master/smac/tae/execute_func.py#L69

[stheid]

can confirm that using lambda istead of partial is indeed another successful workaround.

Thanks for your blazing fast replies. I assume with the pending changes in the code this issue can be assumed closed.

[mfeurer]

Kind of, but could you please leave it open so we know that we need to update the doc?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[timruhkopf]

Have been looking into this. The way you would want to do this in smac 2.0, is that you no longer define the tae as a method (because the partial will treat the method as static) but instead do it like this:

def train(config: Configuration, seed: int = 0, X=None, y=None) -> float:
    with warnings.catch_warnings():
        warnings.filterwarnings("ignore", category=ConvergenceWarning)
        
        classifier = MLPClassifier(
            hidden_layer_sizes=[config["n_neurons"]] * config["n_layer"],
            solver=config["optimizer"],
            batch_size=config["batch_size"],
            activation=config["activation"],
            learning_rate_init=config["learning_rate_init"],
            random_state=seed,
            max_iter=5,
        )

        # Returns the 5-fold cross validation accuracy
        cv = StratifiedKFold(n_splits=5, random_state=seed,
                             shuffle=True)  # to make CV splits consistent
        score = cross_val_score(classifier, X, y, cv=cv, error_score="raise")

        return 1 - np.mean(score)

Now you can do the following:

 from functools import partial

    digits = load_digits()
    tae = partial(train, X=digits.data, y=digits.target)

    tae.__code__ = tae.func.__code__  # this is a crucial step, because the facade takes the code
    # into its meta information and partial does not provide it automatically.

and you are set.