Marius Lindauer

To provide some pointers whether a PIAC approach could be better than plain AC, you could provide some heterogeneity measures. 1. See: http://ml.informatik.uni-freiburg.de/papers/12-LION-hoosch12a.pdf 2. Compute oracle performance - incumbent performance....

Hi, We often wonder how hard an AutoML problem is. Can we therefore add some metrics regarding that? For example * an eCDF plot for the cost distribution (i.e., a...

Plots over time for mf with trajectories of each config from syne tune paper/ video

Many of our analyses are based on surrogate models. It would be fairly important to know how faithful these surrogate models actually are. Could we add some insights regarding that?...

to imitate the original implementation we probably would need to discretize the pcs first, apply a one-hot encoding afterwards and then we can learn the linear model.

Broken fANOVA

3

Running on the spear-qcp example (from SMAC3) ``` $ python ~/git/ParameterImportance/scripts/evaluate.py --scenario_file smac3-output_2017-05-12_12\:53\:31_\(750603\)_run1/scenario.txt --history smac3-output_2017-05-12_12\:53\:31_\(750603\)_run1/runhistory.json --trajectory smac3-output_2017-05-12_12\:53\:31_\(750603\)_run1/traj_aclib2.json --num_params 10 --modus all [...] INFO:fANOVA:PREPROCESSING PREPROCESSING PREPROCESSING PREPROCESSING PREPROCESSING PREPROCESSING INFO:fANOVA:Finished Preprocessing...

Running influence-model on the spear_qcp example, no combinations of parameters are considered. Are there no dependencies between the parameters or is not yet supported?

Predictions in Log-Cost-Space

3

I hope that I correctly remember our discussion yesterday about the predictions in log-cost space since I forgot my notes in my office. @frank-hutter if anything is wrong, please correct...

Please add a function to get the predicted mean and variance for each tree.