workflows icon indicating copy to clipboard operation
workflows copied to clipboard

Add a butcher argument to the model-based calibration methods

Open topepo opened this issue 1 year ago • 2 comments

This will require a change to probably, which does the butchering.

Originally posted by @topepo in #65

topepo avatar Jan 10 '25 17:01 topepo

By "argument" do you mean we should add an argument to axe_*.workflow methods to toggle whether to additionally butcher the calibration objects or just that butchering a workflow ought to butcher the calibration objects generally?

simonpcouch avatar Jan 10 '25 17:01 simonpcouch

The first option: When we butcher the workflow, we should be able to exempt the fitted calibration model (from lm() or gam()) from being trimmed.

topepo avatar Jan 10 '25 20:01 topepo

Currently, nothing of the post-processor gets butchered. Since workflows basically just pull out the relevant objects and lets their own butcher methods do the work, it makes sense to add butcher methods for tailors to tailor https://github.com/tidymodels/tailor/issues/97 first

hfrick avatar Aug 11 '25 19:08 hfrick

@topepo The lm and gam calibration models already get butchered in probably, as part of cal_estimate_linear() and cal_estimate_logistic(), so a toggle here wouldn't exempt them. Do you want this to be made optional in probably, or for the workflows toggle to only affect calibration models for isotonic and multinom? (Beta calibration also gets butchered in probably.)

For reference:

  • cal_estimate_linear(): https://github.com/tidymodels/probably/blob/a30e1a27370cf5d55b27cbf881afb78a27387712/R/cal-estimate-linear.R#L180-L196
  • cal_estimate_logistic(): https://github.com/tidymodels/probably/blob/a30e1a27370cf5d55b27cbf881afb78a27387712/R/cal-estimate-logistic.R#L165-L179

hfrick avatar Aug 14 '25 10:08 hfrick

Do you want this to be made optional in probably, or for the workflows toggle to only affect calibration models for isotonic and multinom?

Yes, the first step is to update probably. tailor is recently able to pass ... to probably so people can stop the butchering via tailor. I'll open an issue in probably.

I don't believe the isotonic methods need butchering since they basically hold a lookup table. The multinomial methods should be able to be butchered.

topepo avatar Aug 14 '25 13:08 topepo

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.

github-actions[bot] avatar Aug 29 '25 01:08 github-actions[bot]