Proposal: Sweepable API

[LittleLittleCloud]

This is the initial design/proposal doc for a Sweepable API.

I would appreciate it if you could each review it and give me your feedback.

[codecov[bot]]

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@b31b7ca). Click here to learn what that means. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #5993   +/-   ##
=======================================
  Coverage        ?   68.54%           
=======================================
  Files           ?     1146           
  Lines           ?   245450           
  Branches        ?    25637           
=======================================
  Hits            ?   168240           
  Misses          ?    70502           
  Partials        ?     6708           

Flag	Coverage Δ
Debug	68.54% <0.00%> (?)
production	63.32% <0.00%> (?)
test	88.67% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

[torronen]

From user perspective this looks great and promising! As I understand pipeline.SearchSpace will automatically keep track of the experiments, their results and parameters. So, basically, we could start each new tuning experiment by first running the best parameters from last experiment, then looping with tuner.Propose. Is it correct?

If possible, I'd also like to extract the results from pipeline.SearchSpace to help me get a better understanding of how the tuning affects performance. For example, I might want to plot a graph about how the number of trees affect FastForest to help me find optimal number considering training time vs. accuracy.

[LittleLittleCloud]

From user perspective this looks great and promising! As I understand pipeline.SearchSpace will automatically keep track of the experiments, their results and parameters. So, basically, we could start each new tuning experiment by first running the best parameters from last experiment, then looping with tuner.Propose. Is it correct?

@torronen Quite close but not exactly accurate. Pipeline.SearchSpace will only define the boundary of parameters and and it's stateless. If you want to track parameter and their results under current api, you need to do it yourself. (save each parameter and their result in a dictionary or something..)

If possible, I'd also like to extract the results from pipeline.SearchSpace to help me get a better understanding of how the tuning affects performance. For example, I might want to plot a graph about how the number of trees affect FastForest to help me find optimal number considering training time vs. accuracy.

That's a great advice. It's also something that we want to heavily dive into on how to help users understand training process, like trend line of metric, importance of each parameter, etc etc...

[torronen]

@LittleLittleCloud ok I see. I assume Tuner needs to keep the results.

Something in general, the sweeping process is running through a loop. So it's very flexible on how you perform training and you can customize it into whatever way you want, like hold-out strategy, metric, or even training on multiple thread/on cloud...

// psudo code
// Input: Tuner, SearchSpace, SweepablePipeline, trainDataset, testDataset
foreach (var parameter in tuner.Propose())
{
   var mlnetPipeline = SweepablePipeline.BuildTrainingPipeline(parameter)

   // there're really no limit on how to train your model, the following example uses hold-out strategy
   var model = mlnetPipeline.Fit(trainDataset)
   var eval = model.Transform(testDataset)
   
   // there're also no limit on how to calculate your metric, you can either use built-in optimize metric of define your own
   var metric = calculateYourMetric(eval, truth)

   // for some tuners they need to be aware of metric so as to propose the possible best parameter next time.
   tuner.Update(parameter, metric)
}

Not sure if these are valid for this plan, but my next questions might be:

How to change the optimization metric?

optimization metric is a number calculated from predict and truth value so as long as you have access to predict and truth value (which is available) you can calculate whatever metric you want.

How to create custom optimization metric? See https://github.com/dotnet/machinelearning/issues/5999 Maybe derived from other metrics (maybe as anonymous method?), or even create a mini-simulation. Based on my initial experiments with simulating real-life events, simulation results might be more sensitive to tuning than traditional metrics. For example, we may want to catch some of different types of malicious web requests to catch the bad actors. Same ML metrics may come for detecting 100% of all requests for a single type, but in this case we might only catch one bad actor. In a simulation we might want to see how early intruders are detected: it's not enough to detect intruder when the logout.

If ML.Net built-in optimization metric can't meet your request, you can define your own metric and calculate by yourself as long as you have predict and truth value. ML.Net DataFrame api might be helpful in this case.

How to create a new Tuner? Not sure if necessary. It can also be done without being this type of Tuner. Could be also one area where to make it easy for community to contribute their implementations as extension nugets, or even as part of Microsoft.ML if dev team thinks it is feasible.

You can implement ITuner interface to create your own tuner.

On a higher level API it will be not that easy to customize the entire training process. A fluent-style will be preferred here

var experiemnt = new AutoMLExperiment()
                               .AddPipeline(sweepablePipeline)
                               .AddTuner(tuner)
                               .AddMetric(async (predict, actual) => {// return metric})
                               .Train();

But you'll still have access to all low-level api so there'll be a way to customize the training.

[LittleLittleCloud]

@torronen reply inline

[torronen]

@LittleLittleCloud BTW, does this proposal and search space API allow optimizing the booster in LightGBM? Or, should users run the tuning for DART, Gradient and Goss separately is they expect it to matter? Asking because it is an object, not value type, and the object has parameters inside it.

Or, maybe user can write his own if...else or switch depending on what the tuner wants to run next?

[LittleLittleCloud]

@LittleLittleCloud BTW, does this proposal and search space API allow optimizing the booster in LightGBM? Or, should users run the tuning for DART, Gradient and Goss separately is they expect it to matter? Asking because it is an object, not value type, and the object has parameters inside it.

Or, maybe user can write his own if...else or switch depending on what the tuner wants to run next?

Yes you can, it really doesn't matter whether the property you want to sweep on is a value type or not, you can aways pass a factory method to create the trainer based on parameter sweeped from search space.

[michaelgsharp]

@LittleLittleCloud Merging this in as the sweepable estimator itself is already checked in.