[Question] My Question?

[Estefano13]

I'm trying to produce learning curves for a couple of models. the models were initially trained and saved using pickle. After loading and refit on the whole subset, I experience no issues. However, when attempting to refit the models on training data subsets, the refit process fails due to: ValueError: SelectClassificationRates removed all features.

My data is an array of floats with no negative values. Because I am trying to get scores at different training set sizes, the training subsets sometimes have more features than samples (Could this be responsible for the problem?). The model was originally fitted on the whole training set using GroupKFold as cv strategy.

I've observed that this issue somewhat depends on the subset being trained on, with a bias toward failing toward smaller subsets. Here is the error output:

ValueError Traceback (most recent call last) in <cell line: 1>() ----> 1 learning_curve_AutoML(my_model_, X_train.to_numpy(), y_train, groups = X_amplitude_train["Sample ID"], cv=5, scoring = "balanced_accuracy", train_sizes=np.linspace(.3, 1.0, 5))

11 frames in learning_curve_AutoML(estimator_, X, y, groups, train_sizes, scoring, cv, ret_data) 36 37 print(X_train_0.shape) ---> 38 estimator_.refit(X_train_0, y_train_0) 39 40 train_scores = score(estimator, X_train_0, y_train_0, scorer)

/usr/local/lib/python3.10/dist-packages/autosklearn/estimators.py in refit(self, X, y) 792 793 """ --> 794 self.automl_.refit(X, y) 795 return self 796

/usr/local/lib/python3.10/dist-packages/autosklearn/automl.py in refit(self, X, y, max_reshuffles) 1214 1215 if i == (max_reshuffles - 1): -> 1216 raise e 1217 1218 self._can_predict = True

/usr/local/lib/python3.10/dist-packages/autosklearn/automl.py in refit(self, X, y, max_reshuffles) 1194 try: 1195 if self._budget_type is None: -> 1196 _fit_and_suppress_warnings(self._logger, model, X, y) 1197 else: 1198 _fit_with_budget(

/usr/local/lib/python3.10/dist-packages/autosklearn/evaluation/abstract_evaluator.py in _fit_and_suppress_warnings(logger, model, X, y) 186 with warnings.catch_warnings(): 187 warnings.showwarning = send_warnings_to_log --> 188 model.fit(X, y) 189 190 return model

/usr/local/lib/python3.10/dist-packages/autosklearn/pipeline/base.py in fit(self, X, y, **fit_params) 122 a classification algorithm first. 123 """ --> 124 X, fit_params = self.fit_transformer(X, y, **fit_params) 125 self.fit_estimator(X, y, **fit_params) 126 return self

/usr/local/lib/python3.10/dist-packages/autosklearn/pipeline/classification.py in fit_transformer(self, X, y, fit_params) 121 fit_params.update(_fit_params) 122 --> 123 X, fit_params = super().fit_transformer(X, y, fit_params=fit_params) 124 125 return X, fit_params

/usr/local/lib/python3.10/dist-packages/autosklearn/pipeline/base.py in fit_transformer(self, X, y, fit_params) 134 } 135 fit_params_steps = self._check_fit_params(**fit_params) --> 136 Xt = self._fit(X, y, **fit_params_steps) 137 return Xt, fit_params_steps[self.steps[-1][0]] 138

/usr/local/lib/python3.10/dist-packages/sklearn/pipeline.py in _fit(self, X, y, **fit_params_steps) 301 cloned_transformer = clone(transformer) 302 # Fit or load from cache the current transformer --> 303 X, fitted_transformer = fit_transform_one_cached( 304 cloned_transformer, X, y, None, 305 message_clsname='Pipeline',

/usr/local/lib/python3.10/dist-packages/joblib/memory.py in call(self, *args, **kwargs) 310 311 def call(self, *args, **kwargs): --> 312 return self.func(*args, **kwargs) 313 314 def call_and_shelve(self, *args, **kwargs):

/usr/local/lib/python3.10/dist-packages/sklearn/pipeline.py in _fit_transform_one(transformer, X, y, weight, message_clsname, message, **fit_params) 754 res = transformer.fit_transform(X, y, **fit_params) 755 else: --> 756 res = transformer.fit(X, y, **fit_params).transform(X) 757 758 if weight is None:

/usr/local/lib/python3.10/dist-packages/autosklearn/pipeline/components/feature_preprocessing/select_rates_classification.py in transform(self, X) 94 95 if Xt.shape[1] == 0: ---> 96 raise ValueError("%s removed all features." % self.class.name) 97 return Xt 98

ValueError: SelectClassificationRates removed all features.

System Details

I am running this on Google Colab

python version: 3.10.12 Autosklearn version = 0.15.0 sklearn version = 0.24.2

Is this working as intended? Any suggestions as to how to avoid this problem in the future? Should I just exclude the problematic feature preprocessing step?