auto-sklearn icon indicating copy to clipboard operation
auto-sklearn copied to clipboard

Published 20 hours ago • verified publisher icon automl

[Request] Allow portfolio and selector models to be set through hyperparameters in ASKL2

Open PGijsbers opened this issue 4 years ago • 4 comments

As per the title, it will be useful for ASKL2 to have a configurable portfolio and policy selector. It's beneficial for research (avoiding 'cheating' through meta-learning in a benchmark) or for customization.

Issue opened on the request of @mfeurer

PGijsbers avatar Jul 30 '21 13:07 PGijsbers

Hi @PGijsbers, this would certainly be a nice feature and definitely something to have in once asklearn 2 becomes non-experimental. I had a quick look to see how possible this is and I don't imagine it's too bad depending on the definition of configurable for policy selector and portifolio.

Simply changing the portifolio file location and enabling disabling it should be quite straightforward and likewise with white listing or black listing certain policys. However if there is a need for injecting a custom policy selector object it might be a bit more involved ... but also manageable.

Do you need this for the automlbenchmark and do you have a time urgency for this?

eddiebergman avatar Aug 04 '21 11:08 eddiebergman

Yes it's something we'd ultimately like to use in the AutoML benchmark. Though realistically I don't think we can wait for this to be all done and set up properly before doing our next set of experiments, which have been delayed for months already.

PGijsbers avatar Aug 16 '21 09:08 PGijsbers

No problem, seeing as this will probably not be done in time for the benchmarks, I will talk to @mfeurer about how we do this. Perhaps autosklearn2 will become the default before the next benchmark run.

eddiebergman avatar Aug 16 '21 10:08 eddiebergman

Hi, We have been using autosklearn and automlbenchmark in R&D at SAS Institiute for autoML pipeline. Huge applauses to the great work of you all first! @eddiebergman @PGijsbers

I have a few questions regarding portfolio.

  1. I am trying to replicate the portfolio building in the autosklearn2 paper. Is these any code shared in this repo? I search the portfolio file but I couldn't find it. Maybe I have don't have the permission to the version that you are referring to.
  2. For my current research project, I'd like to add/delete portfolio. Is this supported? If not, I am glad to contribute by adding this feature.

Thanks, Yonglin

zhuygln avatar Oct 11 '22 15:10 zhuygln