Error stacking classification models with `pr_auc`

[cgoo4]

Hi - Stacks is proving very effective at improving model performance for me (using the output from finetune::tune_sim_anneal()). Thank you!

I am occasionally running into this problem though stacking classification models with metric pr_auc.

It's a bit of a contrived example to attempt to recreate the error:

(I also seem to get it when num_members > 0.)

library(tidymodels)
library(stacks)

data("tree_frogs")

tree_frogs <- tree_frogs |> 
  select(-c(clutch, latency)) |> 
  mutate(reflex = if_else(row_number() < 20, "low", "other"))

set.seed(1)

tree_frogs_split <- initial_split(tree_frogs)
tree_frogs_train <- training(tree_frogs_split)
tree_frogs_test  <- testing(tree_frogs_split)

folds <- vfold_cv(tree_frogs_train, v = 5)

tree_frogs_rec <- 
  recipe(reflex ~ ., data = tree_frogs_train) |> 
  step_dummy(all_nominal_predictors(), -reflex) |> 
  step_zv(all_predictors())

tree_frogs_wflow <- 
  workflow() |> 
  add_recipe(tree_frogs_rec)

ctrl_grid <- control_stack_grid()

rand_forest_spec <- 
  rand_forest(
    mtry = tune(),
    min_n = tune(),
    trees = 500
  ) %>%
  set_mode("classification") |> 
  set_engine("ranger")

rand_forest_wflow <-
  tree_frogs_wflow |> 
  add_model(rand_forest_spec)

rand_forest_res <- 
  tune_grid(
    object = rand_forest_wflow, 
    resamples = folds, 
    grid = 10,
    control = ctrl_grid
  )
#> i Creating pre-processing data to finalize unknown parameter: mtry

nnet_spec <-
  mlp(hidden_units = tune(), penalty = tune(), epochs = tune()) |> 
  set_mode("classification") |> 
  set_engine("nnet")

nnet_rec <- 
  tree_frogs_rec |> 
  step_normalize(all_predictors())

nnet_wflow <- 
  tree_frogs_wflow |> 
  add_model(nnet_spec) |> 
  update_recipe(nnet_rec)

nnet_res <-
  tune_grid(
    object = nnet_wflow, 
    resamples = folds, 
    grid = 10,
    control = ctrl_grid
  )

tree_frogs_model_st <- 
  stacks() |> 
  add_candidates(rand_forest_res) |> 
  add_candidates(nnet_res) |> 
  blend_predictions(
    metric = metric_set(pr_auc),
    mixture = 1,
    penalty = seq(0, 1, 0.1),
    ) |> 
  fit_members()
#> → A | warning: one multinomial or binomial class has fewer than 8  observations; dangerous ground
#> There were issues with some computations   A: x1
#> There were issues with some computations   A: x3
#> 

autoplot(tree_frogs_model_st)
#> Warning in ggplot2::scale_x_log10(): log-10 transformation introduced infinite values.
#> log-10 transformation introduced infinite values.

length(tree_frogs_model_st$member_fits)
#> [1] 0

tree_frogs_model_st |> 
  augment(tree_frogs_test, type = "prob")
#> Error in `tidyr::pivot_wider()`:
#> ! Can't select columns past the end.
#> ℹ Location 3 doesn't exist.
#> ℹ There are only 2 columns.

^{Created on 2024-08-17 with reprex v2.1.1}

[simonpcouch]

Thanks for the detailed issue description! This a funky one.

In general, I think the best we can do here is just supply a really informative error from predict() and augment() informing folks that there are no members and they'll need to reduce penalty or set it 0. That said:

I also seem to get it when num_members > 0.

If you're able to provide a reproducible example, I'd be very much interested in checking it out! From what I can tell, the symptom is the zero member fits, but I'm very much open to being shown otherwise. :)

[cgoo4]

Hi Simon - Would it be possible for stacks to ignore the 0-member solutions if there are other valid solutions?

With my real data some mixture / penalty combinations had valid solutions with multiple members, but because the best pr_auc happened to have 0 members (which may be a separate issue on my side!) it didn't select a useable solution.

I can get there by adjusting the parameter combinations and re-running, but what makes that tricky is that num_members may be > 0 in the autoplot whilst length(x$member_fits) = 0, so it's not obvious from the plot what will fail. This is an example with my real data:

> length(class_stack$member_fits)
[1] 0

This was the example I was hoping I could minimally reprex.

[simonpcouch]

Would it be possible for stacks to ignore the 0-member solutions if there are other valid solutions?

Interesting thought--thanks for the suggestion. I definitely missed some other potential solutions here.

One is as you propose, where we choose a sub-optimal solution that results in some members being trained. As I re-read this, though, this feels more like a situation where we just train the optimal model—which happens to be intercept-only and incorporates predictions from no members. The fact that we hadn't done so already feels more like a software bug to me.