why is prob required for auc?

[tdhock]

hi I am getting an unexpected warning when I try to use ctree with an AutoTuner,

mlr3tuning::AutoTuner$new(
    learner = mlr3extralearners::lrn("classif.ctree", predict_type="prob"),
    resampling = mlr3::rsmp("cv", folds = 3),
    measure = mlr3::msr("classif.auc"),
    search_space = paradox::ps(
      alpha = paradox::p_dbl(lower = 0, upper = 1)
    ),
    terminator = mlr3tuning::trm("none"),
    tuner = mlr3tuning::tnr("grid_search", resolution = 5),
    store_tuning_instance = TRUE
  )

I get

Warning message:
Measure 'classif.auc' is missing predict type 'prob' of learner 'classif.ctree' 

Probabilities are not required to compute ROC AUC, so why is this warning happening? I would expect that any classifier that outputs a real-valued score can be used with ROC AUC.

In this case the warning goes away if you specify predict_type="prob" argument of lrn,

mlr3tuning::AutoTuner$new(
    learner = mlr3extralearners::lrn("classif.ctree", predict_type="prob"),
    resampling = mlr3::rsmp("cv", folds = 3),
    measure = mlr3::msr("classif.auc"),
    search_space = paradox::ps(
      alpha = paradox::p_dbl(lower = 0, upper = 1)
    ),
    terminator = mlr3tuning::trm("none"),
    tuner = mlr3tuning::tnr("grid_search", resolution = 5),
    store_tuning_instance = TRUE
  )

[sebffischer]

I would expect that any classifier that outputs a real-valued score can be used with ROC AUC.

But if the response type is set to "response" the outputs is not real valued but the predicted labels

[mllg]

I agree that the name "probabilities" is a bit misleading, as they are often not real probabilities. Do you know a learner which we have tagged to not be able to predict probabilities, but which is capable of scoring the predictions?