Nested cross-validation

[zachmayer]

trafficstars

It'd be nice to evaluate the ensemble on a separate test set than the one used to make the ensemble.

[jknowles]

:+1: Absolutely agree with this.

[rlesca01]

I wrote a basic script a couple functions to do this a few weeks ago (rlesca01/caretNestedCV). Its pretty rough at the moment but it does work. Basically it just does a loop over a function which is very similar to caretList and returns a list of a list of models. There is then a function calculates the OOS performance of the individual models and then another function which returns the OOS of the ensembled models. The main.r script should just run as is with the example data. Obviously, if you're doing 10x CV on the outer loop than everything takes much longer.

[zachmayer]

@rlesca01 If you want to incorporate this into caretEnsemble, feel free to send a PR.

Otherwise, Jared or I will give it a shot soonish.

[rlesca01]

Hi Zach, yeah I'll take a stab at this and send over a PR, mostly I need to add some tests to the code I have.