[ENH] `sktime` time series regression integration

[fkiraly]

It would be nice to have an estimator level integration with time series regressors in the sktime package. https://github.com/sktime/sktime

This would require:

• a BaseExperiment descendant class SktimeRegressionExperiment which carries out a cross-validation experiment for time series regressors. This can lean on SktimeClassificationExperiment, in particular when handling metrics. In hyperactive.experiment.integrations. Also see the extension template https://github.com/SimonBlanke/Hyperactive/blob/main/extension_templates/experiments.py
• a TSROptCv class inheriting from sktime BaseRegressor similar to the other classes in hyperactive.integrations. Internally, it uses the SktimeRegressionExperiment

[Khushmagrawal]

Hey @fkiraly, I would like to work on this issue. I have gone through the existing code and PRs related to integration (#173 & #200 ). Please let me know if I should know anything else

[fkiraly]

Great! It is very similar to time series classification.

Once this is added to hyperactive, there is a second setp to add a record in sktime.

[Khushmagrawal]

Hey @fkiraly, While implementing the SktimeRegressionExperiment, following the SktimeClassificationExperiment design, I tried locating corresponding evaluate utility for regression. I went through some related issues https://github.com/sktime/sktime/issues/6852 & https://github.com/sktime/sktime/issues/6708) and PRs (https://github.com/sktime/sktime/pull/6888 & https://github.com/sktime/sktime/pull/8267)

From the discussion, it appeared that an evaluate utility would be introduced for both TSC and TSR, However, in the PRs I only see the evaluate function implemented under: sktime.classification.model_evaluation and there doesnt seem to be a similar sktime.regression.model_evaluation

Should TSR also have a evaluate function similar to the TSC or does evaluation utility for regression already exist elsewhere and I may be overlooking it?

[fkiraly]

Hm, interesting question.

Currently, if I look under the hood, there is nothing that seems to prevent us from using the classification evaluate for regression. Could you check if this is the case? If so, then I would do the following:

• modify the evaluate docstring in sktime that clearly states that it also allows to use regressors
• add a folder in regression that mirrors the one in classification, but it only imports the existing evaluate
• modify the docstring of the experiment in hyperactive that it also covers regressors

In both cases, some thinking may have to go into the default score - determining whether the estimator is a classifier or regressor can be done using the object_type tag.

However, if there are good reasons why a separate evaluate may be needed, I would suggest to first discuss here then.

[Khushmagrawal]

I think the core logic is largely similar, but combining regression and classification into a single evaluate function would require few if/else branches to handle cases like avoiding predict_proba (classification-only) for regressors and setting different default scoring as you mentioned.

Given that models are kept isolated in sktime, it is consistent for the evaluation functions to follow the same structure and remain separate. While this introduces some DRY tradeoff, the resulting code would be more understandable

[Khushmagrawal]

I think the core logic is largely similar, but combining regression and classification into a single evaluate function would require few if/else branches to handle cases like avoiding predict_proba (classification-only) for regressors and setting different default scoring as you mentioned.

Given that models are kept isolated in sktime, it is consistent for the evaluation functions to follow the same structure and remain separate. While this introduces some DRY tradeoff, the resulting code would be more understandable

Hey @Omswastik-11, I noticed #209 integrates sktime regressors by reusing classification evaluate utility. I am planning to work on adding this regression-side evaluate in sktime, but wanted to check first if you are already working on it so as to avoid duplication

[Omswastik-11]

Hi @Khushmagrawal !! Thanks for pointing out. I was also wandering why there is no separate evaluation utility for Sktime regression so I tried with classification . For now it seems correct. I will dig more into it and let you know if that can help you writing a new evaluate for Sktime regression. I was thinking to adding this regression-side evaluate in sktime but you can proceed with this. Plz ping me when you done with changes I will modify the Sktime regression integration accordingly .