makeClassifTask "positive" argument does not change positive class

[hrj21]

Hi all, I know mlr is no longer actively maintained, but in case it's an easy fix, the positive argument to the makeClassifTask() function doesn't seem to do anything. Here is a reprex showing that the only way to change the positive class is to change the order of the factor levels manually beforehand.

library(mlr)
library(tidyverse)
data(titanic_train, package = "titanic")
titanicTib <- as_tibble(titanic_train)
fctrs <- c("Survived", "Sex", "Pclass")
titanicClean <- titanicTib %>%
  mutate_at(.vars = fctrs, .funs = factor) %>%
  mutate(FamSize = SibSp + Parch) %>%
  select(Survived, Pclass, Sex, Age, Fare, FamSize)

logReg <- makeLearner("classif.logreg", predict.type = "prob")

imp <- impute(titanicClean, cols = list(Age = imputeMean()))
titanicTask <- makeClassifTask(data = imp$data, target = "Survived")
logRegModel <- train(logReg, titanicTask)

titanicTask1 <- makeClassifTask(data = imp$data, target = "Survived", positive = "1")
logRegModel1 <- train(logReg, titanicTask1)

getLearnerModel(logRegModel)
getLearnerModel(logRegModel1)

imp$data$Survived <- factor(imp$data$Survived, levels = c("1", "0"))
titanicTask2 <- makeClassifTask(data = imp$data, target = "Survived")
logRegModel2 <- train(logReg, titanicTask2)
a<-getLearnerModel(logRegModel2)
logRegModel2$factor.levels
logRegModel2$task.desc

Session info:

sessionInfo()
R version 4.0.1 (2020-06-06)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server 7.4 (Maipo)

Matrix products: default
BLAS:   /opt/R/4.0.1/lib64/R/lib/libRblas.so
LAPACK: /opt/R/4.0.1/lib64/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8   
 [6] LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] forcats_0.5.1     stringr_1.4.0     dplyr_1.0.7       purrr_0.3.4       readr_2.0.1       tidyr_1.1.3       tibble_3.1.3     
 [8] ggplot2_3.3.5     tidyverse_1.3.1   mlr_2.19.0        ParamHelpers_1.14

loaded via a namespace (and not attached):
 [1] tidyselect_1.1.1  splines_4.0.1     haven_2.4.0       lattice_0.20-41   colorspace_2.0-2  vctrs_0.3.8       generics_0.1.0   
 [8] XML_3.99-0.7      utf8_1.2.2        survival_3.1-12   rlang_0.4.11      pillar_1.6.2      withr_2.4.2       glue_1.4.2       
[15] DBI_1.1.1         dbplyr_2.1.1      modelr_0.1.8      readxl_1.3.1      lifecycle_1.0.0   munsell_0.5.0     gtable_0.3.0     
[22] cellranger_1.1.0  rvest_1.0.0       parallelMap_1.5.0 tzdb_0.1.2        parallel_4.0.1    fansi_0.5.0       broom_0.7.9      
[29] Rcpp_1.0.7        scales_1.1.1      backports_1.2.1   checkmate_2.0.0   jsonlite_1.7.2    fs_1.5.0          fastmatch_1.1-3  
[36] hms_1.1.0         stringi_1.7.3     BBmisc_1.11       grid_4.0.1        cli_3.0.1         tools_4.0.1       magrittr_2.0.1   
[43] crayon_1.4.1      pkgconfig_2.0.3   ellipsis_0.3.2    Matrix_1.2-18     xml2_1.3.2        data.table_1.14.0 reprex_2.0.0     
[50] lubridate_1.7.10  rstudioapi_0.13   assertthat_0.2.1  httr_1.4.2        R6_2.5.1          compiler_4.0.1  

[pat-s]

'positive' is only stored as a metadata in the task, it is not used during training AFACIS. I think it might only be used during scoring but I did not dig fully into it.

If it would not be used at all this would be a huge issue. Given our immense test suite this is unlikely though.

On a side note: is there anything you're missing from mlr3 that makes you still use mlr?

[berndbischl]

no, its used in (at least) 2 places.

1. is is used in ROC metrics, where you need this for TP, FP and so on. this is what patrick referred to.

2. but it is also used in the encoding in the learner itself. whether or not thats done is the resposibility of the code in the learner. let me quickly check

[berndbischl]

see here

https://github.com/mlr-org/mlr/blob/main/R/RLearner_classif_logreg.R

line 34 p = as.factor(ifelse(x > 0.5, levs[2L], levs[1L]))

used "positive" as levs = c(negative, positive)

[berndbischl]

the problem now, and we recently saw this in in mlr3 as well, is that this ensures that the predictions and all metrics are correct.

but if you look at the internal state of the model, and directions of effects, they can be reversed. and i completely agree that this is uncool and should be fixed

[berndbischl]

@pat-s if you want to fix this, you can compare here, where it was already fixed https://github.com/mlr-org/mlr3learners/blob/main/R/LearnerClassifLogReg.R

and no, sadly, this is not easily detectable by our unit tests - which we indeed have a lot

1. first of all, one can talk about whether predictions and scores are correct - or "label swapping" occurs. here, i am reasonably sure, that we are good for all learners that we have. but note: a) you have to take care of that for every individual learners. and this is really a pain in the ass in R. every package does this differently. its often not really documented, etc. OTOH i tried to come up with a general test for all learners. the situation still sucks a bit.

2. what the OP here is referring to, is something thats reflected in the model after you have left mlr. which of course still matters. for this you cannot even write a general test.