caretEnsemble icon indicating copy to clipboard operation
caretEnsemble copied to clipboard

Improve prediction intervals and documentation

Open jknowles opened this issue 10 years ago • 3 comments

This is a four part task:

  • [ ] Create a more robust prediction interval for predict.caretEnsemble
  • [ ] Apply this method to predict.caretStack to allow it to produce intervals
  • [ ] Document prediction interval calculation and its interpretation
  • [ ] Create a fall-back for ensembles that include models that fail to produce valid predictions

I am not sure how to handle part 4 just yet.

jknowles avatar May 21 '15 15:05 jknowles

Hmmm, part 4 is tricky.

Part 2 is tricky as well. Some caretStack models (e.g. glm) will have prediction intervals, and others (e.g. gbm) will not.

zachmayer avatar May 21 '15 15:05 zachmayer

Yes, Part 2 is really tricky if we want to include uncertainty within each model into the equation. Currently on the caretEnsemble side it just incorporates the disagreement between models within the library -- which is not enough.

Also, the performance of predict in both cases is pretty slow :turtle: so I'll want to check into that too.

jknowles avatar May 21 '15 15:05 jknowles

Sounds good!

zachmayer avatar May 21 '15 16:05 zachmayer