ValueError in `CircularValidation` Scorer with Deep Models

[YanisLalou]

Description

When using the CircularValidation scorer with deep models, a ValueError is raised during the inverse transformation of predicted source labels. This error occurs because the predicted labels (y_pred_source) contain previously unseen labels that are not present in the original labels (y).

Traceback

Traceback (most recent call last):
  File "/mnt/beegfs/workdir/yanis.lalou/anaconda3/envs/env_8_7_24/lib/python3.10/site-packages/sklearn/metrics/_scorer.py", line 141, in __call__
    score = scorer(estimator, *args, **routed_params.get(name).score)
  File "/mnt/beegfs/workdir/yanis.lalou/anaconda3/envs/env_8_7_24/lib/python3.10/site-packages/skada/metrics.py", line 44, in __call__
    return self._score(estimator, X, y, sample_domain=sample_domain, **params)
  File "/mnt/beegfs/workdir/yanis.lalou/anaconda3/envs/env_8_7_24/lib/python3.10/site-packages/skada/metrics.py", line 649, in _score
    y_pred_source = le.inverse_transform(y_pred_source)
  File "/mnt/beegfs/workdir/yanis.lalou/anaconda3/envs/env_8_7_24/lib/python3.10/site-packages/sklearn/preprocessing/_label.py", line 160, in inverse_transform
    raise ValueError("y contains previously unseen labels: %s" % str(diff))
ValueError: y contains previously unseen labels: [63]

Dataset/Method to Reproduce

Dataset: OfficeHome Method: Deep_NO_DA_SOURCE_ONLY

Additional Context

This issue is likely to occur with deep models where the model can output labels that are not necessarily present in the training data.