themis
themis copied to clipboard
Published
20 hours ago •
tidymodels
tidymodels
Missing values in importance weights
trafficstars
The problem
I'm working with a dataset where I use importance weights to specify the misclassification costs of instances. Because the target class in the dataset is severly unbalanced, I would like to use some resample (e.g. SMOTE) to mitigate this issue. However, step_smote() and friends do not impute the importance weights, and I cannot impute them later because this is not allowed by other step_impute methods.
I can understand that the default behaviour should not be to generate new weights as well as this might lead to unexpected behaviour, but I do not see why the algorithms in this package would be unable to do this at all.
Reproducible example
Here is an example using hpc_data:
library(tidymodels)
library(themis)
# First, get rid of the nominal predictors as these cannot be used by `step_smote`
hpc_data <- select(hpc_data, -c(protocol, day))
# Now specify the importance weights, for example input_fields
hpc_data <- mutate(hpc_data, input_fields = importance_weights(input_fields))
# Specify a simple recipe to use with `step_smote`
rec <- recipe(class ~ ., data = hpc_data) |>
step_smote(class)
# Now prep and bake as training data to see the result
rec |>
prep() |>
bake(NULL)
#> # A tibble: 8,844 × 6
#> compounds input_fields iterations num_pending hour class
#> <dbl> <imp_wts> <dbl> <dbl> <dbl> <fct>
#> 1 997 137 20 0 14 F
#> 2 97 103 20 0 13.8 VF
#> 3 101 75 10 0 13.8 VF
#> 4 93 76 20 0 10.1 VF
#> 5 100 82 20 0 10.4 VF
#> 6 100 82 20 0 16.5 VF
#> 7 105 88 20 0 16.4 VF
#> 8 98 95 20 0 16.7 VF
#> 9 101 91 20 0 16.2 VF
#> 10 95 92 20 0 10.8 VF
#> # ℹ 8,834 more rows
# This would leave us with 8844 rows, but there are many missing values in input_fields
rec |>
prep() |>
bake(NULL) |>
drop_na(input_fields) # Only 4331 rows left, the same amount as the original dataset
#> # A tibble: 4,331 × 6
#> compounds input_fields iterations num_pending hour class
#> <dbl> <imp_wts> <dbl> <dbl> <dbl> <fct>
#> 1 997 137 20 0 14 F
#> 2 97 103 20 0 13.8 VF
#> 3 101 75 10 0 13.8 VF
#> 4 93 76 20 0 10.1 VF
#> 5 100 82 20 0 10.4 VF
#> 6 100 82 20 0 16.5 VF
#> 7 105 88 20 0 16.4 VF
#> 8 98 95 20 0 16.7 VF
#> 9 101 91 20 0 16.2 VF
#> 10 95 92 20 0 10.8 VF
#> # ℹ 4,321 more rows
# On the other hand, `step_upsample()` does work
rec <- recipe(class ~ ., data = hpc_data) |>
step_upsample(class)
rec |>
prep() |>
bake(NULL) |>
drop_na(input_fields)
#> # A tibble: 8,844 × 6
#> compounds input_fields iterations num_pending hour class
#> <dbl> <imp_wts> <dbl> <dbl> <dbl> <fct>
#> 1 97 103 20 0 13.8 VF
#> 2 101 75 10 0 13.8 VF
#> 3 93 76 20 0 10.1 VF
#> 4 100 82 20 0 10.4 VF
#> 5 100 82 20 0 16.5 VF
#> 6 105 88 20 0 16.4 VF
#> 7 98 95 20 0 16.7 VF
#> 8 101 91 20 0 16.2 VF
#> 9 95 92 20 0 10.8 VF
#> 10 102 96 20 0 9.97 VF
#> # ℹ 8,834 more rows
Created on 2024-04-15 with reprex v2.1.0
Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.3.3 (2024-02-29 ucrt)
#> os Windows 10 x64 (build 19045)
#> system x86_64, mingw32
#> ui RTerm
#> language (EN)
#> collate English_United Kingdom.utf8
#> ctype English_United Kingdom.utf8
#> tz Europe/Brussels
#> date 2024-04-15
#> pandoc 3.1.1 @ C:/Workdir/MyApps/RStudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> package * version date (UTC) lib source
#> backports 1.4.1 2021-12-13 [1] CRAN (R 4.1.2)
#> broom * 1.0.5 2023-06-09 [1] CRAN (R 4.3.1)
#> class 7.3-22 2023-05-03 [1] CRAN (R 4.3.0)
#> cli 3.6.2 2023-12-11 [1] CRAN (R 4.3.2)
#> codetools 0.2-19 2023-02-01 [1] CRAN (R 4.2.2)
#> colorspace 2.1-0 2023-01-23 [1] CRAN (R 4.2.2)
#> data.table 1.15.0 2024-01-30 [1] CRAN (R 4.3.2)
#> dials * 1.2.1 2024-02-22 [1] CRAN (R 4.3.2)
#> DiceDesign 1.10 2023-12-07 [1] CRAN (R 4.3.2)
#> digest 0.6.34 2024-01-11 [1] CRAN (R 4.3.2)
#> dplyr * 1.1.4 2023-11-17 [1] CRAN (R 4.3.2)
#> ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.0.5)
#> evaluate 0.23 2023-11-01 [1] CRAN (R 4.3.2)
#> fansi 1.0.6 2023-12-08 [1] CRAN (R 4.3.2)
#> fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.2.2)
#> foreach 1.5.2 2022-02-02 [1] CRAN (R 4.1.3)
#> fs 1.6.3 2023-07-20 [1] CRAN (R 4.3.1)
#> furrr 0.3.1 2022-08-15 [1] CRAN (R 4.2.1)
#> future 1.33.1 2023-12-22 [1] CRAN (R 4.3.2)
#> future.apply 1.11.1 2023-12-21 [1] CRAN (R 4.3.2)
#> generics 0.1.3 2022-07-05 [1] CRAN (R 4.2.1)
#> ggplot2 * 3.5.0 2024-02-23 [1] CRAN (R 4.3.2)
#> globals 0.16.2 2022-11-21 [1] CRAN (R 4.2.2)
#> glue 1.7.0 2024-01-09 [1] CRAN (R 4.3.2)
#> gower 1.0.1 2022-12-22 [1] CRAN (R 4.2.2)
#> GPfit 1.0-8 2019-02-08 [1] CRAN (R 4.0.0)
#> gtable 0.3.4 2023-08-21 [1] CRAN (R 4.3.1)
#> hardhat 1.3.1 2024-02-02 [1] CRAN (R 4.3.2)
#> htmltools 0.5.7 2023-11-03 [1] CRAN (R 4.3.2)
#> infer * 1.0.6 2024-01-31 [1] CRAN (R 4.3.2)
#> ipred 0.9-14 2023-03-09 [1] CRAN (R 4.2.2)
#> iterators 1.0.14 2022-02-05 [1] CRAN (R 4.1.3)
#> knitr 1.45 2023-10-30 [1] CRAN (R 4.3.2)
#> lattice 0.22-5 2023-10-24 [1] CRAN (R 4.3.2)
#> lava 1.7.3 2023-11-04 [1] CRAN (R 4.3.2)
#> lhs 1.1.6 2022-12-17 [1] CRAN (R 4.2.2)
#> lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.3.1)
#> listenv 0.9.1 2024-01-29 [1] CRAN (R 4.3.2)
#> lubridate 1.9.3 2023-09-27 [1] CRAN (R 4.3.2)
#> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.1.3)
#> MASS 7.3-60.0.1 2024-01-13 [1] CRAN (R 4.3.2)
#> Matrix 1.6-5 2024-01-11 [1] CRAN (R 4.3.2)
#> modeldata * 1.3.0 2024-01-21 [1] CRAN (R 4.3.2)
#> munsell 0.5.0 2018-06-12 [1] CRAN (R 4.0.0)
#> nnet 7.3-19 2023-05-03 [1] CRAN (R 4.3.0)
#> parallelly 1.37.1 2024-02-29 [1] CRAN (R 4.3.2)
#> parsnip * 1.2.0 2024-02-16 [1] CRAN (R 4.3.2)
#> pillar 1.9.0 2023-03-22 [1] CRAN (R 4.2.3)
#> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.0.0)
#> prodlim 2023.08.28 2023-08-28 [1] CRAN (R 4.3.2)
#> purrr * 1.0.2 2023-08-10 [1] CRAN (R 4.3.1)
#> R.cache 0.16.0 2022-07-21 [1] CRAN (R 4.2.1)
#> R.methodsS3 1.8.2 2022-06-13 [1] CRAN (R 4.2.0)
#> R.oo 1.26.0 2024-01-24 [1] CRAN (R 4.3.2)
#> R.utils 2.12.3 2023-11-18 [1] CRAN (R 4.3.2)
#> R6 2.5.1 2021-08-19 [1] CRAN (R 4.1.1)
#> RANN 2.6.1 2019-01-08 [1] CRAN (R 4.0.0)
#> Rcpp 1.0.12 2024-01-09 [1] CRAN (R 4.3.2)
#> recipes * 1.0.10 2024-02-18 [1] CRAN (R 4.3.2)
#> reprex 2.1.0 2024-01-11 [1] CRAN (R 4.3.2)
#> rlang 1.1.3 2024-01-10 [1] CRAN (R 4.3.2)
#> rmarkdown 2.25 2023-09-18 [1] CRAN (R 4.3.2)
#> ROSE 0.0-4 2021-06-14 [1] CRAN (R 4.3.3)
#> rpart 4.1.23 2023-12-05 [1] CRAN (R 4.3.2)
#> rsample * 1.2.0 2023-08-23 [1] CRAN (R 4.3.1)
#> rstudioapi 0.15.0 2023-07-07 [1] CRAN (R 4.3.1)
#> scales * 1.3.0 2023-11-28 [1] CRAN (R 4.3.2)
#> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.1.2)
#> styler 1.10.2 2023-08-29 [1] CRAN (R 4.3.1)
#> survival 3.5-8 2024-02-14 [1] CRAN (R 4.3.2)
#> themis * 1.0.2 2023-08-14 [1] CRAN (R 4.3.3)
#> tibble * 3.2.1 2023-03-20 [1] CRAN (R 4.2.3)
#> tidymodels * 1.1.1 2023-08-24 [1] CRAN (R 4.3.1)
#> tidyr * 1.3.1 2024-01-24 [1] CRAN (R 4.3.2)
#> tidyselect 1.2.1 2024-03-11 [1] CRAN (R 4.3.3)
#> timechange 0.3.0 2024-01-18 [1] CRAN (R 4.3.2)
#> timeDate 4032.109 2023-12-14 [1] CRAN (R 4.3.2)
#> tune * 1.1.2 2023-08-23 [1] CRAN (R 4.3.1)
#> utf8 1.2.4 2023-10-22 [1] CRAN (R 4.3.2)
#> vctrs 0.6.5 2023-12-01 [1] CRAN (R 4.3.2)
#> withr 3.0.0 2024-01-16 [1] CRAN (R 4.3.2)
#> workflows * 1.1.4 2024-02-19 [1] CRAN (R 4.3.2)
#> workflowsets * 1.0.1 2023-04-06 [1] CRAN (R 4.2.3)
#> xfun 0.42 2024-02-08 [1] CRAN (R 4.3.2)
#> yaml 2.3.8 2023-12-11 [1] CRAN (R 4.3.2)
#> yardstick * 1.3.0 2024-01-19 [1] CRAN (R 4.3.2)
#>
#> [1] C:/Workdir/MyApps/R-Library/4.0
#> [2] C:/Workdir/MyApps/R/R-4.3.3/library
#>
#> ──────────────────────────────────────────────────────────────────────────────
Proposed solution
I'm wondering whether this behaviour can be implemented in the functions in this work package, if necessary not as the default behaviour. If there is some other solution that I've missed, I'd be more than happy to learn more about it.
Hello @koenniem 👋
The main reason why steps such as step_smote() doesn't work with importance weights, is because there isn't information as to how the weights should be inputed.
Propose for example that the importance weight is a measure of oldness. how should step_smote() fill in the weights? there is no assumption that the weights have any relation to the predictors. So it is doing the best it can and fill in with NA.
If you have prior knowledge, you could use step_mutate(w = if_else(is.na(w), importance_weights(52), w)) but you should be very careful when doing it.
And honestly, you would be better off using step_upsample() or step_downsample() as they work with weights.
