DeepCAVE icon indicating copy to clipboard operation
DeepCAVE copied to clipboard

New plugin: Hardness of AutoML problem

Open mlindauer opened this issue 3 years ago • 0 comments

Hi,

We often wonder how hard an AutoML problem is. Can we therefore add some metrics regarding that? For example

  • an eCDF plot for the cost distribution (i.e., a hard AutoML task should have only a few very well-performing configurations)
  • uni-modal metric from the automl loss landscape paper (but evaluated on our surrogate models)
  • convexity metric from the automl loss landscape paper (but evaluated on our surrogate models)

mlindauer avatar Jul 21 '22 09:07 mlindauer