FR: Add option to control_* functions to do a test run on only one resample to check for warnings/errors

[joranE]

Feature

In situations when I'm exploring multiple preprocessors & models it can be really frustrating to run code such as:

results <- 
  # set of workflows with possibly multiple preprocessors & models
  wfl_set |> 
  workflow_map(
    resamples = cv_folds, 
    fn = "tune_grid", 
    metrics = metric_set(roc_auc, accuracy, precision)  
)

...and have it chug through 90% of the combinations & folds only to hit some combination late in the run where all models fail because I've made a mistake in my recipe or engine definitions.

Obviously, it's possible to test each recipe & model spec individually before tossing them all at the parameter tuning, but that can involve quite a bit more work & code that you ultimately end up writing more or less only for the purpose of verifying that everything merely runs without errors (or warnings).

I feel like it might be a significant time saver if there were an option in functions like control_grid, eg control_grid(preliminary = TRUE) that ran each workflow on only a single resample and a single parameter combination merely for the purposes of validating that the preprocessor & models run without error.

I realize that this won't catch 100% of potential errors due to weird variations in either the cv folds or the parameter combinations, but I feel like it has the potential to significantly speed up the loop of writing code, running, finding problems and fixing them.

[simonpcouch]

Ah, interesting. Related to #722.

I do feel like a control argument in each control_*() function is more appealing to me as an interface than a separate pilot_*() function for each tuning function. Less clutter in the namespace and less code redundancy.

[joranE]

Oh, weird, that related issue was apparently prompted by a mastodon post by me? This is an odd variation on searching for how to do something and landing on my own SO answer.

[EmilHvitfeldt]

I agree @simonpcouch! This feels ergonomically very similar to turning on and off verbose

[simonpcouch]

This blog post makes some points that might support adding this feature: https://www.mm218.dev/posts/2024-07-19-tidymodels/