auto-sklearn icon indicating copy to clipboard operation
auto-sklearn copied to clipboard

Published 20 hours ago • verified publisher icon automl

callback function error!

Open Daniel-Trung-Nguyen opened this issue 2 years ago • 3 comments

I was trying to apply the early stopping example for a regression problem and got this error:

[ERROR] [2022-08-18 04:33:27,482:Client-AutoML(1):botanal] Cannot register callback of type <class 'function'>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/autosklearn/automl.py", line 931, in fit
    _proc_smac.run_smbo()
  File "/usr/local/lib/python3.7/dist-packages/autosklearn/smbo.py", line 496, in run_smbo
    smac.register_callback(self.trials_callback)
  File "/usr/local/lib/python3.7/dist-packages/smac/facade/smac_ac_facade.py", line 850, in register_callback
    raise ValueError("Cannot register callback of type %s" % type(callback))
ValueError: Cannot register callback of type <class 'function'>
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-30-b2f7d005a0d7>](https://localhost:8080/#) in <module>
      8                                                         get_trials_callback=callback
      9                                                     )
---> 10 automl.fit(X_train, y_train, dataset_name="botanal")

5 frames
[/usr/local/lib/python3.7/dist-packages/smac/facade/smac_ac_facade.py](https://localhost:8080/#) in register_callback(self, callback)
    848                 break
    849         if key is None:
--> 850             raise ValueError("Cannot register callback of type %s" % type(callback))
    851         self.solver._callbacks[key].append(callback)

ValueError: Cannot register callback of type <class 'function'>

from smac.optimizer.smbo import SMBO
from smac.runhistory.runhistory import RunInfo, RunValue
def callback(
    smbo: SMBO,
    run_info: RunInfo,
    result: RunValue,
    time_left: float):
    """Stop early if we get a very low cost value for a single run
    The return value indicates to SMAC whether to stop or not. False will
    stop the search process while any other value will mean it continues.
    """
    # You can find out the parameters in the SMAC documentation
    # https://automl.github.io/SMAC3/main/
    if result.cost <= 0.02:
        print("Stopping!")
        print(run_info)
        print(result)
        return False

automl = autosklearn.regression.AutoSklearnRegressor(time_left_for_this_task=120,
                                                        per_run_time_limit=30,
                                                        tmp_folder="/tmp/autosklearn_regression_example_tmp",
                                                        get_trials_callback=callback
                                                    )
automl.fit(X_train, y_train, dataset_name="botanal")

Daniel-Trung-Nguyen avatar Aug 18 '22 04:08 Daniel-Trung-Nguyen

Hi @Daniel-Trung-Nguyen,

I'll look into this but this feature is currently only in the documentation for development and only available in the development branch, are you using the development branch?

Best, Eddie

eddiebergman avatar Aug 18 '22 08:08 eddiebergman

I just pulled the dev brand and tested the callback (on Colab). I received this error:

Process pynisher function call:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/sklearn/utils/__init__.py", line 236, in _determine_key_type
    return dtype_to_str[type(key)]
KeyError: <class 'numpy.str_'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pynisher/limit_function_call.py", line 133, in subprocess_func
    return_value = ((func(*args, **kwargs), 0))
  File "/usr/local/lib/python3.7/dist-packages/autosklearn/smbo.py", line 166, in _calculate_metafeatures_encoded
    logger=logger_,
  File "/usr/local/lib/python3.7/dist-packages/autosklearn/metalearning/metafeatures/metafeatures.py", line 1122, in calculate_all_metafeatures_encoded_labels
    logger=logger,
  File "/usr/local/lib/python3.7/dist-packages/autosklearn/metalearning/metafeatures/metafeatures.py", line 1194, in calculate_all_metafeatures
    X_transformed = DPP.fit_transform(X)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/base.py", line 699, in fit_transform
    return self.fit(X, **fit_params).transform(X)
  File "/usr/local/lib/python3.7/dist-packages/autosklearn/pipeline/components/data_preprocessing/feature_type.py", line 216, in fit
    self.column_transformer.fit(X, y)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/compose/_column_transformer.py", line 470, in fit
    self.fit_transform(X, y=y)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/compose/_column_transformer.py", line 505, in fit_transform
    self._validate_remainder(X)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/compose/_column_transformer.py", line 325, in _validate_remainder
    for cols in self._columns)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/compose/_column_transformer.py", line 325, in <genexpr>
    for cols in self._columns)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/utils/__init__.py", line 257, in _determine_key_type
    key_type = {_determine_key_type(elt) for elt in unique_key}
  File "/usr/local/lib/python3.7/dist-packages/sklearn/utils/__init__.py", line 257, in <setcomp>
    key_type = {_determine_key_type(elt) for elt in unique_key}
  File "/usr/local/lib/python3.7/dist-packages/sklearn/utils/__init__.py", line 238, in _determine_key_type
    raise ValueError(err_msg)
ValueError: No valid specification of the columns. Only a scalar, list or slice of all integers or all strings, or boolean mask is allowed
[WARNING] [2022-08-18 20:51:10,640:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:11,533:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:12,428:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:13,310:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:14,193:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:15,077:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:15,958:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:16,838:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:17,715:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:18,622:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:19,513:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:21,184:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:22,095:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:23,008:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:23,918:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:24,826:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:25,732:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:26,632:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:27,532:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:28,437:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:29,333:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:30,229:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:31,137:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:32,025:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:32,935:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:33,827:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:39,824:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:45,627:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:47,198:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:55,572:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:57,108:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:58,088:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:03,259:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:04,243:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:05,215:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:10,838:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:11,774:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:12,709:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:13,671:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:19,213:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:20,152:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:21,108:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:27,076:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:28,018:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:28,959:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:29,908:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:30,868:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:31,872:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:32,824:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:33,803:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:34,742:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:35,685:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:36,635:Client-EnsembleBuilder] No runs were available to build an ensemble from

Daniel-Trung-Nguyen avatar Aug 18 '22 20:08 Daniel-Trung-Nguyen

@Daniel-Trung-Nguyen, this seems unrelated but at least it seems that the callback works as intended.

Can you provide a code snippet that reproduces this error, or a snippet of one row of the dataset or at least the feature types of your columns?

Best, Eddie

eddiebergman avatar Aug 19 '22 13:08 eddiebergman