automl
Cave + BOHB: Error in fanova
Hey Joshua,
thanks for your work on Cave!
While using it, I have stumbled upon some errors related to BOHB and Categorical HP. I am pretty sure they are not super relevant and I was able to find a workaround. But I thought it might be useful for you.
My ConfigSpace contains a categorical hyperparameter so running
cave $(pwd) --output cave_output_dir
threw the error which is described in #192
Full Error Message
NFO:fanova.visualizer.Visualizer:creating cave_report_2/analysis_data/-2146368420622879532/fanova/lr_weight_decay.png
ERROR:cave.cavefacade.CAVE:Error in fANOVA - please run with --pimp_no_fanova_pairs (this might be due to a known issue with ints and bools in categorical hyperparameters, see issue #192).
Traceback (most recent call last):
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 324, in cave_fanova
fanova = Fanova(self.runscontainer)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/analyzer/parameter_importance/fanova.py", line 35, in __init__
self.parameter_importance("fanova")
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/analyzer/parameter_importance/base_parameter_importance.py", line 60, in parameter_importance
'bokeh': components(run.pimp.evaluator.plot_bokeh(show_plot=False))
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/pimp/evaluator/fanova.py", line 265, in plot_bokeh
mean, std, grid = vis.generate_marginal(param_name, 100)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/fanova/visualizer.py", line 276, in generate_marginal
(m, v) = self.fanova.marginal_mean_variance_for_values(dim, [grid[i]])
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/fanova/fanova.py", line 367, in marginal_mean_variance_for_values
sample[dimlist[i]] = valuesToPredict[i]
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 55, in wrap
analyzer = f(self, *args, **kw)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 327, in cave_fanova
raise IndexError("Error in fANOVA - please run with --pimp_no_fanova_pairs (this might be due to a known "
IndexError: Error in fANOVA - please run with --pimp_no_fanova_pairs (this might be due to a known issue with ints and bools in categorical hyperparameters, see issue #192).
Traceback (most recent call last):
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 324, in cave_fanova
fanova = Fanova(self.runscontainer)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/analyzer/parameter_importance/fanova.py", line 35, in __init__
self.parameter_importance("fanova")
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/analyzer/parameter_importance/base_parameter_importance.py", line 60, in parameter_importance
'bokeh': components(run.pimp.evaluator.plot_bokeh(show_plot=False))
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/pimp/evaluator/fanova.py", line 265, in plot_bokeh
mean, std, grid = vis.generate_marginal(param_name, 100)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/fanova/visualizer.py", line 276, in generate_marginal
(m, v) = self.fanova.marginal_mean_variance_for_values(dim, [grid[i]])
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/fanova/fanova.py", line 367, in marginal_mean_variance_for_values
sample[dimlist[i]] = valuesToPredict[i]
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/muelleph/miniconda3/envs/operation/bin/cave", line 8, in <module>
sys.exit(entry_point())
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cave_cli.py", line 372, in entry_point
cave.main_cli()
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cave_cli.py", line 297, in main_cli
cave.analyze()
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/utils/timing.py", line 11, in wrap
result = f(*args, **kw)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 233, in analyze
self.parameter_importance(self._get_dict(self.website, "Parameter Importance"))
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 353, in parameter_importance
self.cave_fanova(d=d)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 55, in wrap
analyzer = f(self, *args, **kw)
File "/home/muelleph/miniconda3/envs/operation/lib/python3.6/site-packages/cave/cavefacade.py", line 327, in cave_fanova
raise IndexError("Error in fANOVA - please run with --pimp_no_fanova_pairs (this might be due to a known "
IndexError: Error in fANOVA - please run with --pimp_no_fanova_pairs (this might be due to a known issue with ints and bools in categorical hyperparameters, see issue #192).
However, using the parameter --pimp_no_fanova_pairs did not change anything.
Skipping the entire fanova package has worked. (using the cmd line argument --skip fanova)
I did a clean install of CAVE with (pip install --upgrade cave): cave==1.4.0 PyImp==1.1.2
Thanks for reporting. I'll probably have capacities to look into this in the second half of March (sorry, there is a number of deadlines before), maybe I can help out then.
In the meantime if it's important to you (or someone stumbling upon the issue), it's probably not due to #192 since you're using .json-files. You can still try to cut out individual parameters (maybe the none value of the transform parameter? Wouldn't know why but then again I'm currently not sure where the parsing happens and how), especially categorical hyperparameters were prone to break stuff in fANOVA iirc...
