yardstick icon indicating copy to clipboard operation
yardstick copied to clipboard

Feature request: custom `direction` in `new_metric()`

Open mikemahoney218 opened this issue 2 years ago • 1 comments

Feature

Apologies if this already exists and I've missed it.

It would be useful to allow custom direction options when creating new metrics. My current use-case for this behavior is implementing Geary's C in waywiser, which has an optimal value of 1; while this case is easy to engineer around (a metric C* exists such that C* = C - 1, letting direction = "zero") I think there may be other metrics which would benefit from being able to provide their own direction arguments.

mikemahoney218 avatar Jul 06 '22 20:07 mikemahoney218

The biggest issue with a completely custom direction is that tune somehow has to know how to use it. In particular, tune_bayes() does different things depending on whether you are minimizing/maximizing your metric

https://github.com/tidymodels/tune/blob/e4b7485f6196c005d457f713c2bc5da5ffbfb63c/R/tune_bayes.R#L216

So does show_best() I believe

It wouldn't know how to handle a completely custom direction without some kind of constraints around it.

DavisVaughan avatar Aug 25 '22 14:08 DavisVaughan