MLJ.jl icon indicating copy to clipboard operation
MLJ.jl copied to clipboard

implement functionality for measures search.

Open OkonSamuel opened this issue 2 years ago • 2 comments

Is your feature request related to a problem? Please describe. It would be great to have built into MLJ a way for searching measures suitable to a given predictions yhat and target y, similar to the model searching functionality already implemented.

Describe the solution you'd like So I propose something along these lines, measures(matching(yhat, y)), which would return a list of measures that can be used suitable for predictions yhat and target y.

OkonSamuel avatar Jul 19 '22 20:07 OkonSamuel

Okay this design won't do because of ambiguity with models(matching(X, y)). Also I don't like the fact that matching is forcing us to couple model search code with measure search code. Measures will eventually be independent of the whole stack, except StatisticalTraits.jl.

  1. One suggestion is to deprecate predicates as arguments of models and do something like this:
  • models() acts as currently
  • models(str) acts as currently, for str a string or regex
  • models((X, y)) acts like models(matching(X, y)) does now.
  • models(p) falls back to assume p is a predicate (such as matching(X, y)) but this behaviour is deprecated, as not needed (we have filter for that).
  • measures((yhat, y)) finds all measures compatible with (yhat, y)
  1. A breaking alternative is to modify the third item above and allow models(X, y) to behave as models(matching(X, y)) does now, but that would disallow the possibility that X is a string, say, and we would have to disallow predicates immediately.

ablaom avatar Jul 25 '22 22:07 ablaom

Another nice enhancement would be to allow kwargs for searching over trait values, as in models(package_name="ScikitLearn", is_supervised=true). Currently we have the ugly

models() do m
    m.package_name(m) == ScikitLearn" &&
    m.is_supervised
end

ablaom avatar Aug 23 '22 00:08 ablaom