caret
caret copied to clipboard
Custom summary measure with OOB resampling?
I am trying to use caret
to fit a random forest binary classification model using OOB
resampling with a custom summary measure.
However, I get the following warning:
library(caret);
### custom performance metric function
# https://stackoverflow.com/a/52697940
mySummary <- function (data, lev = NULL, model = NULL) {
a1 <- defaultSummary(data, lev, model) # accuracy, kappa
b1 <- twoClassSummary(data, lev, model) # area under ROC, sens, spec
c1 <- prSummary(data, lev, model) # area under PR curve, prec, recall, F-score
out <- c(a1, b1, c1);
return(out);
}
# training options
seed = 123
train.ctrl.rf <- trainControl(
method = "oob",
classProbs = TRUE,
summaryFunction = mySummary
);
Warning message: Custom summary measures cannot be computed for out-of-bag resampling. This value of
summaryFunction
will be ignored.
Is there any plan to support custom summary measures with OOB
resampling?