defining information on tuning parameters "in place"

[simonpcouch]

We recently discussed possible alternative interfaces for defining information on tuning parameters.

Changes to make this happen would happen in many packages, though the main context where folks would encounter this change is in tune, so filing the issue here.

Some setup:

library(tidymodels)

mtcars <- tibble(mtcars[rep(1:32, 5),])

xgb_v_fold <- 
  vfold_cv(data = mtcars)

xgb_recipe <- 
  recipe(mtcars, formula = mpg ~ cyl + disp)

Currently, users tag an argument for tuning, and if they want to update the, say, range associated with a tuning parameters, they'd extract_parameter_set_dials() the specification/recipe/workflow, update() the parameter set, and then pass it along with the specification/recipe/workflow to the tuning function. This looks like:

xgb_mod <- 
  boost_tree("regression", min_n = tune())

xgb_param_set <-
  extract_parameter_set_dials(xgb_mod) %>%
  update(min_n = min_n(range = c(1, 5)))

xgb_results <- 
  workflow() %>% 
  add_model(xgb_mod) %>% 
  add_recipe(xgb_recipe) %>% 
  tune_grid(resamples = xgb_v_fold, param_info = xgb_param_set)

I find this a bit cumbersome, and somewhat counter to the first stated goal of workflows, that "You don’t have to keep track of separate objects in your workspace." It also leads to an occasional gotcha where folks update a workflow specification and forget to update the parameter set object that was derived from it.

I would propose that we ought to look into somehow associating parameter tuning information with the object itself (not necessarily as a parameter set).

a)

One interface could involve supporting dials objects supplied directly as model arguments. In pseudocode:

xgb_mod <- 
  boost_tree("regression", min_n = min_n(range = c(1, 5)))

xgb_results <- 
  workflow() %>% 
  add_model(xgb_mod) %>% 
  add_recipe(xgb_recipe) %>% 
  tune_grid(resamples = xgb_v_fold)

This doesn't mean that a parameter set is now associated with xgb_mod---just that we have additional information to pass as update arguments when tune passes xgb_mod to extract_parameter_set_dials() internally.

One technical challenge here is figuring out how to pass a unique id (as in the argument to tune()).

b)

Another possible interface would be to introduce ellipses to tune() that are passed to the dials function associated with the model argument.

xgb_mod <- 
  boost_tree("regression", min_n = tune(range = c(1, 5)))

xgb_results <- 
  workflow() %>% 
  add_model(xgb_mod) %>% 
  add_recipe(xgb_recipe) %>% 
  tune_grid(resamples = xgb_v_fold)

From a backend standpoint, this is somewhat complex, but from a user's standpoint, I think this is just as eliciting as ^^^.

c)

One last option could be to introduce a helper that mirror's update()'s functionality, but works "in place" on a model specification (or recipe, or workflow).

xgb_mod <- 
  boost_tree(min_n = tune()) %>%
  update_parameter(min_n = min_n(range = c(1, 5)))

xgb_results <- 
  workflow() %>% 
  add_model(xgb_mod) %>% 
  add_recipe(xgb_recipe) %>% 
  tune_grid(resamples = xgb_v_fold)

In this case, it's nice that tune() doesn't need any changes, but update_parameter() feels like it's handling something that the model argument ought to be taking care of.

---

In weighing these (or additional) options, I'd make the argument that technical effort involved in implementing them doesn't matter. Any of these proposals are quite tractable technically, and I'd say we ought to focus on the interface improvement for users.

^{Created on 2023-08-02 with reprex v2.0.2}

[EmilHvitfeldt]

We should also think about how we handle wrongly specified tuning parameters. The following is obviously nonsense,

boost_tree("regression", min_n = threshold(range = c(0, 0.5)))

But we can't match with names directly, as there are some places where the dials object is different than the argument name

recipe(mpg ~ ., data = mtcars) |>
  step_poly(disp, degree = degree_int())

[simonpcouch]

Yup, absolutely. We can match to the dials function just via tunable() methods, though. For min_n, via the model registration and tunable.model_spec(), and for recipes arguments like step_poly(degree), in the tunable methods themselves, which makes checking much more tractable.

[topepo]

I like option (b) (see my alternate option (b+) below).

There are a few technical difficulties.

First, we are capturing the inputs and not evaluating them so we might have to carry around some quosures. From recipe experiences, we probably don’t want to keep the quosures around for long.

For non-numeric tuning parameters, people will probably not use c() in place to make the list of values. We'll probably have to evaluate the arguments quickly or deal with more global data.

Second, what would the argument signature look like? Currently, it just captures the (character) value (i.e. tune(id = "")). If we add ellipses, we could require that all other arguments be named. This means that users could still do

   step_pca(x, y, x, num_terms = tune("pca", range = c(1, 20)))

but we would need to protect against:

   step_pca(x, y, x, num_terms = tune(c(1, 20)))

so that we don’t get id = “c(1, 20)” (which is legal).

Option (b+)

Do the same as (b), but use a synonym for "tune". That would allow us backward compatibility and an alternate argument signature.

Maybe opt (for optimize), vary() (NOT varying() 😄 ), candidate(), param(), or something else. Maybe dial() is too on the nose.

Alternatively, we could be the first to have a function with an emoji name 🎵 = function(...) (jk)

If we used ... as the argument signature, we could perhaps have more leeway and not worry about the legacy id argument.

[simonpcouch]

Good point on the id argument, and I think your proposal to a transition to ... could be a good workaround. Agreed that something b)-ish might feel the most expressive—will toss these ideas around when implementing and see if there are any unexpected complications that might make b or b+ more principled.

[simonpcouch]

Somewhat of a change of heart here in that it seems possibly most likely that a user would know that they'd like to change parameter ranges after tuning once, at which the current API doesn't require revisiting the model spec itself but this API would. In terms of ease of use for users, this seems like a lateral move, and would be a good bit of development energy on our part and additional complexity (2 interfaces to choose from) on the part of users. Will close unless others feel this is worth pursuing.