Temporal CV

[pfistfl]

I currently have a task with a column that is a date. As the task is to basically predict values in the future, a cross-validation strategy that can take this into account would be required. Similar to see RollingWindowCV. As this is a very common use-case, we should perhaps think about implementing this.

• This is implemented in mlr3forecasting, but for forecasting tasks instead of regular Classif|Regr Tasks.
• Where should such a method live? mlr3spatiotempcv ?
• How would we go about implementing this.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label feature_request to this issue, with a confidence of 0.56. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[pat-s]

A standalone temporal CV would definitely fit here, yes.

One can take different approaches of accounting for this, the most common is probably clustering (with kmeans as the default approach). Other approaches I know are predefined groups of temporal clusters (if the groups are clearly separated).

The latter is already doable, the kmeans clustering can be quickly adopted by spcv-coords.

In the end you want to ensure to decluster observations that are close in time because they show a high correlation among them naturally. This is the same issue as observations in space.

And yes, we can port over RollingWindowCV.

[pfistfl]

I think the thing we want here is not clustering, but instead basically splitting train / test such, that max(train$date) < min(test$date), i.e. we always test how well our algorithms generate to future settings. This means the train data grows in each fold, just as RollingWindowCV.

I think @mllg wanted this as well, have you already started something there?

[pat-s]

Ah ok, this is also an interesting approach!

In RollingWindowCV, if you specify folds, you discard some obs in some folds, is that correct? (judging from the example and test fold 1).

I am not sure if folds is a good name here since usually folds are characterized to be unique in the test sets - this is more a bootstrapping approach? Would iters be a better term in this case?

An argument supporting a percentage increase could be interesting?

have you already started something there?

Nope, nothing exists in this way, never had such a dataset yet. But I'd say it would fit really good into this package.

[pfistfl]

Datasets:

• The bikesharing dataset is an example of such a dataset, we use it in two gallery posts: https://github.com/mlr-org/mlr3gallery/pull/13 https://github.com/mlr-org/mlr3gallery/pull/64

A ressource on time-series cross-validation: https://robjhyndman.com/hyndsight/tscv/ Here they call it fold, but i do not care a lot about the naming.

[mllg]

I believe this would also fit nicely in mlr3. Tasks already have column role "order" which can be used in something like "ResamplingOrderedCV" or "ResamplingOrderedHoldout".

[pat-s]

If we have already a dedicated package for spatial and temporal CV stuff, I'd argue it should live there, simply because users might look for it there?

[pat-s]

@pfistfl

Coming back to this after a while, I now have a different view on this:

• I think it would be neat if we would have one dedicated package to spatiotemporal tasks and resampling methods and I think {mlr3spatiotempcv} would be a good fit. Also I think having different tasks classes is more confusing than it helps as spatial or temporal tasks share many properties. Thus, TaskRegrST and TaskClassifST already have temporal in their name.
• I see {mlr3forecasting} more on the same level as {mlr3raster}, i.e. taking care of the prediction calls while leaving task and resampling to {mlr3spatiotempcv}

From a user point of view, task and resampling stuff could then be done with one extension package (i.e. {mlr3spatiotempcv}. When it comes to prediction/measures/learners, {mlr3raster} (or maybe {mlr3spatial}) and {mlr3forecasting} would come into play.

Thoughts?

[pat-s]

Oliveira et al 2021 could be an interesting read.

[pat-s]

I think I would like to postpone the implementation after the paper has been submitted. Including it before would require to introduce and discuss a somewhat distinct field which I would like to avoid right now.

[ck37]

I need this kind of method to use mlr3 for EHR-based machine learning - specifically the ability to define training/test/validation sets using date-based splits.

Is it possible for me to provide the splits to mlr3 and use the existing framework? I wasn't able to see how to do that in the documentation so far. It seems like I will need to use tidymodels otherwise.

[pfistfl]

Hey @ck37

If you are able to compute indices for yourself, you can do it already, see (https://mlr3.mlr-org.com/reference/mlr_resamplings_custom.html).

library(mlr3)
task = tsk("penguins")
task$filter(1:10)

# Instantiate Resampling
custom = rsmp("custom")
train_sets = list(1:5, 5:10)
test_sets = list(5:10, 1:5)
custom$instantiate(task, train_sets, test_sets)

custom$train_set(1)
custom$test_set(1)

[ck37]

Ah ok, awesome - appreciate the help & fast response 🙏