MLOps.NET
MLOps.NET copied to clipboard
Add support to get best run and metrics that are zero-optimized
Is your feature request related to a problem? Please describe.
Our current IsBestRun
method tries to find the best run for a given metric by finding the largest value. This works well for 99% or so of cases but e.g. log loss entropy should be closer to zero to be a good value.
Describe the solution you'd like Add ability to define what a good run looks like for a given metric
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context Add any other context or screenshots about the feature request here.
@aslotte what is your thinking here? To have an overload method for GetBestRun
where the user can request what the metric is based on, e.g closest to 0 (ascending), closest to 1 (descending)? E.g return the log loss entropy (closest to 0). Or something else....
Current - public Run GetBestRun(Guid experimentId, string metricName)
Possible overload - public Run GetBestRun(Guid experimentId, string metricName, int closerTo)
Could also change the current method, or add our own logic, if metric = logloss = ascending etc....