AttributeError: 'LinearSVC' object has no attribute 'predict_proba'

[eddiebergman]

Hi,

I'm using TPOT as part of a paper benchmarking autoML tools and MCS tools (DESlib). I've run into an issue where a pipeline exposes predict_proba as a method but one of the pipeline steps does not actually support this and causes a crash.

Context of the issue

TPOT has generated the following model but the LinearSVC step does not support predict_proba causing an AttributeError: 'LinearSVC' object has no attribute 'predict_proba' when used in further steps, i.e. tpot_classifier.predict_proba(X_test). A further look at sklearn.svm.LinearSVC confirms this to be the case.

tpot_classifier.fit(X, y)

if 'predict_proba' in dir(tpot_classifier): 
    tpot_classifier.predict_proba(X_test)

 AttributeError: 'LinearSVC' object has no attribute 'predict_proba'

This would be fine if the method was not exposed as the library DESlib checks for this but unfortunatly it is exposed without it being implemented throughout the full pipeline.

The full model is:

In[.] tpot_classifier.get_params()
{'memory': Memory(location=/tmp/tmp1ft4o52c/joblib),
 'steps': [('featureunion',
   FeatureUnion(transformer_list=[('featureunion-1',
                                   FeatureUnion(transformer_list=[('functiontransformer-1',
                                                                   FunctionTransformer(func=<function copy at 0x2aaab2389d30>)),
                                                                  ('functiontransformer-2',
                                                                   FunctionTransformer(func=<function copy at 0x2aaab2389d30>))])),
                                  ('featureunion-2',
                                   FeatureUnion(transformer_list=[('functiontransformer-1',
                                                                   FunctionTransformer(func=<function copy at 0x2aaab2389d30>)),
                                                                  ('functiontransformer-2',
                                                                   FunctionTransformer(func=<function copy at 0x2aaab2389d30>))]))])),
  ('stackingestimator-1',
   StackingEstimator(estimator=LinearSVC(C=0.0001, dual=False, random_state=5,
                                         tol=1e-05))),
  ('stackingestimator-2',
   StackingEstimator(estimator=LinearSVC(C=0.001, dual=False, penalty='l1',
                                         random_state=5, tol=1e-05))),
  ('linearsvc', LinearSVC(C=25.0, loss='hinge', random_state=5, tol=0.1))],
 'verbose': False,
 'featureunion': FeatureUnion(transformer_list=[('featureunion-1',
                                 FeatureUnion(transformer_list=[('functiontransformer-1',
                                                                 FunctionTransformer(func=<function copy at 0x2aaab2389d30>)),
                                                                ('functiontransformer-2',
                                                                 FunctionTransformer(func=<function copy at 0x2aaab2389d30>))])),
                                ('featureunion-2',
                                 FeatureUnion(transformer_list=[('functiontransformer-1',
                                                                 FunctionTransformer(func=<function copy at 0x2aaab2389d30>)),
                                                                ('functiontransformer-2',
                                                                 FunctionTransformer(func=<function copy at 0x2aaab2389d30>))]))]),
 'stackingestimator-1': StackingEstimator(estimator=LinearSVC(C=0.0001, dual=False, random_state=5,
                                       tol=1e-05)),
 'stackingestimator-2': StackingEstimator(estimator=LinearSVC(C=0.001, dual=False, penalty='l1',
                                       random_state=5, tol=1e-05)),
 'linearsvc': LinearSVC(C=25.0, loss='hinge', random_state=5, tol=0.1),
 'featureunion__n_jobs': None,
 'featureunion__transformer_list': [('featureunion-1',
   FeatureUnion(transformer_list=[('functiontransformer-1',
                                   FunctionTransformer(func=<function copy at 0x2aaab2389d30>)),
                                  ('functiontransformer-2',
                                   FunctionTransformer(func=<function copy at 0x2aaab2389d30>))])),
  ('featureunion-2',
   FeatureUnion(transformer_list=[('functiontransformer-1',
                                   FunctionTransformer(func=<function copy at 0x2aaab2389d30>)),
                                  ('functiontransformer-2',
                                   FunctionTransformer(func=<function copy at 0x2aaab2389d30>))]))],
 'featureunion__transformer_weights': None,
 'featureunion__verbose': False,
 'featureunion__featureunion-1': FeatureUnion(transformer_list=[('functiontransformer-1',
                                 FunctionTransformer(func=<function copy at 0x2aaab2389d30>)),
                                ('functiontransformer-2',
                                 FunctionTransformer(func=<function copy at 0x2aaab2389d30>))]),
 'featureunion__featureunion-2': FeatureUnion(transformer_list=[('functiontransformer-1',
                                 FunctionTransformer(func=<function copy at 0x2aaab2389d30>)),
                                ('functiontransformer-2',
                                 FunctionTransformer(func=<function copy at 0x2aaab2389d30>))]),
 'featureunion__featureunion-1__n_jobs': None,
 'featureunion__featureunion-1__transformer_list': [('functiontransformer-1',
   FunctionTransformer(func=<function copy at 0x2aaab2389d30>)),
  ('functiontransformer-2',
   FunctionTransformer(func=<function copy at 0x2aaab2389d30>))],
 'featureunion__featureunion-1__transformer_weights': None,
 'featureunion__featureunion-1__verbose': False,
 'featureunion__featureunion-1__functiontransformer-1': FunctionTransformer(func=<function copy at 0x2aaab2389d30>),
 'featureunion__featureunion-1__functiontransformer-2': FunctionTransformer(func=<function copy at 0x2aaab2389d30>),
 'featureunion__featureunion-1__functiontransformer-1__accept_sparse': False,
 'featureunion__featureunion-1__functiontransformer-1__check_inverse': True,
 'featureunion__featureunion-1__functiontransformer-1__func': <function copy.copy(x)>,
 'featureunion__featureunion-1__functiontransformer-1__inv_kw_args': None,
 'featureunion__featureunion-1__functiontransformer-1__inverse_func': None,
 'featureunion__featureunion-1__functiontransformer-1__kw_args': None,
 'featureunion__featureunion-1__functiontransformer-1__validate': False,
 'featureunion__featureunion-1__functiontransformer-2__accept_sparse': False,
 'featureunion__featureunion-1__functiontransformer-2__check_inverse': True,
 'featureunion__featureunion-1__functiontransformer-2__func': <function copy.copy(x)>,
 'featureunion__featureunion-1__functiontransformer-2__inv_kw_args': None,
 'featureunion__featureunion-1__functiontransformer-2__inverse_func': None,
 'featureunion__featureunion-1__functiontransformer-2__kw_args': None,
 'featureunion__featureunion-1__functiontransformer-2__validate': False,
 'featureunion__featureunion-2__n_jobs': None,
 'featureunion__featureunion-2__transformer_list': [('functiontransformer-1',
   FunctionTransformer(func=<function copy at 0x2aaab2389d30>)),
  ('functiontransformer-2',
   FunctionTransformer(func=<function copy at 0x2aaab2389d30>))],
 'featureunion__featureunion-2__transformer_weights': None,
 'featureunion__featureunion-2__verbose': False,
 'featureunion__featureunion-2__functiontransformer-1': FunctionTransformer(func=<function copy at 0x2aaab2389d30>),
 'featureunion__featureunion-2__functiontransformer-2': FunctionTransformer(func=<function copy at 0x2aaab2389d30>),
 'featureunion__featureunion-2__functiontransformer-1__accept_sparse': False,
 'featureunion__featureunion-2__functiontransformer-1__check_inverse': True,
 'featureunion__featureunion-2__functiontransformer-1__func': <function copy.copy(x)>,
 'featureunion__featureunion-2__functiontransformer-1__inv_kw_args': None,
 'featureunion__featureunion-2__functiontransformer-1__inverse_func': None,
 'featureunion__featureunion-2__functiontransformer-1__kw_args': None,
 'featureunion__featureunion-2__functiontransformer-1__validate': False,
 'featureunion__featureunion-2__functiontransformer-2__accept_sparse': False,
 'featureunion__featureunion-2__functiontransformer-2__check_inverse': True,
 'featureunion__featureunion-2__functiontransformer-2__func': <function copy.copy(x)>,
 'featureunion__featureunion-2__functiontransformer-2__inv_kw_args': None,
 'featureunion__featureunion-2__functiontransformer-2__inverse_func': None,
 'featureunion__featureunion-2__functiontransformer-2__kw_args': None,
 'featureunion__featureunion-2__functiontransformer-2__validate': False,
 'stackingestimator-1__estimator__C': 0.0001,
 'stackingestimator-1__estimator__class_weight': None,
 'stackingestimator-1__estimator__dual': False,
 'stackingestimator-1__estimator__fit_intercept': True,
 'stackingestimator-1__estimator__intercept_scaling': 1,
 'stackingestimator-1__estimator__loss': 'squared_hinge',
 'stackingestimator-1__estimator__max_iter': 1000,
 'stackingestimator-1__estimator__multi_class': 'ovr',
 'stackingestimator-1__estimator__penalty': 'l2',
 'stackingestimator-1__estimator__random_state': 5,
 'stackingestimator-1__estimator__tol': 1e-05,
 'stackingestimator-1__estimator__verbose': 0,
 'stackingestimator-1__estimator': LinearSVC(C=0.0001, dual=False, random_state=5, tol=1e-05),
 'stackingestimator-2__estimator__C': 0.001,
 'stackingestimator-2__estimator__class_weight': None,
 'stackingestimator-2__estimator__dual': False,
 'stackingestimator-2__estimator__fit_intercept': True,
 'stackingestimator-2__estimator__intercept_scaling': 1,
 'stackingestimator-2__estimator__loss': 'squared_hinge',
 'stackingestimator-2__estimator__max_iter': 1000,
 'stackingestimator-2__estimator__multi_class': 'ovr',
 'stackingestimator-2__estimator__penalty': 'l1',
 'stackingestimator-2__estimator__random_state': 5,
 'stackingestimator-2__estimator__tol': 1e-05,
 'stackingestimator-2__estimator__verbose': 0,
 'stackingestimator-2__estimator': LinearSVC(C=0.001, dual=False, penalty='l1', random_state=5, tol=1e-05),
 'linearsvc__C': 25.0,
 'linearsvc__class_weight': None,
 'linearsvc__dual': True,
 'linearsvc__fit_intercept': True,
 'linearsvc__intercept_scaling': 1,
 'linearsvc__loss': 'hinge',
 'linearsvc__max_iter': 1000,
 'linearsvc__multi_class': 'ovr',
 'linearsvc__penalty': 'l2',
 'linearsvc__random_state': 5,
 'linearsvc__tol': 0.1,
 'linearsvc__verbose': 0}

Expected result

Either for all generated pipelines to have predict_proba enabled or to remove the exposed method if the pipeline can not support it.

Possible fix

A try/catch on a pipelines predict_proba to determine if it should be exposed or only allow for probabilistic enabled models in a pipeline.

This stackoverflow post suggests a parameter that can be passed to sklearn's svm models to enable probabilistic outputs but is for some reason not enabled for LinearSVC, the thread suggests some further ways to enable this.

Version

> pip list | grep -e TPOT -e sklearn
auto-sklearn     0.12.1
TPOT             0.11.7
> python -V
Python 3.8.6

Python 3.8.6

[sshh12]

Just ran into this, one possible workaround is to just remove it from the search space.

from tpot.config.classifier import classifier_config_dict

new_config = classifier_config_dict.copy()
del new_config['sklearn.svm.LinearSVC']

clf = tpot.TPOTClassifier(config_dict=new_config)