Dynamically pick evaluation strategy

[PGijsbers]

A static 5-fold CV is not good. It should ideally be dynamic ranging from e.g. repeated k-fold for small datasets to hold-out for large ones.

[PGijsbers]

Auto-sklearn 2.0 worked this out (partially). They pick an evaluation strategy based on dataset meta-features, which seems to improve performance (as expected). They still report issues with datasets which are outside their train meta-dataset, adapting evaluation strategy online, especially for these datasets, might be a useful improvement.