MLJ.jl
MLJ.jl copied to clipboard
implement functionality for measures search.
Is your feature request related to a problem? Please describe.
It would be great to have built into MLJ a way for searching measures suitable to a given predictions yhat
and target y
, similar to the model searching functionality already implemented.
Describe the solution you'd like
So I propose something along these lines, measures(matching(yhat, y))
, which would return a list of measures that can be used suitable for predictions yhat
and target y
.
Okay this design won't do because of ambiguity with models(matching(X, y))
. Also I don't like the fact that matching
is forcing us to couple model search code with measure search code. Measures will eventually be independent of the whole stack, except StatisticalTraits.jl.
- One suggestion is to deprecate predicates as arguments of
models
and do something like this:
-
models()
acts as currently -
models(str)
acts as currently, forstr
a string or regex -
models((X, y))
acts likemodels(matching(X, y))
does now. -
models(p)
falls back to assumep
is a predicate (such asmatching(X, y)
) but this behaviour is deprecated, as not needed (we havefilter
for that). -
measures((yhat, y))
finds all measures compatible with(yhat, y)
- A breaking alternative is to modify the third item above and allow
models(X, y)
to behave asmodels(matching(X, y))
does now, but that would disallow the possibility thatX
is a string, say, and we would have to disallow predicates immediately.
Another nice enhancement would be to allow kwargs for searching over trait values, as in models(package_name="ScikitLearn", is_supervised=true)
. Currently we have the ugly
models() do m
m.package_name(m) == ScikitLearn" &&
m.is_supervised
end