Adding individual configuration run detailed output

[phiferd]

This is just a quick POC for the additional details that I thought would be useful to (optionally) output to the user. There are many other things that might be interesting to look at (group by classifier, by feature selection method, etc). Another reason this might be useful is that it gives a better feel for how much of the space was covered. Here's an example of the output:

...
======= Detailed Results ======
1.52 (9.09 / 6): weka.classifiers.functions.SimpleLogistic:[-W, 0, -A], null: [], NONE: []
2.27 (9.09 / 4): weka.classifiers.trees.J48:[-O, -B, -S, -M, 4, -C, 0.18749064322639875], weka.attributeSelection.CfsSubsetEval: [-L], weka.attributeSelection.GreedyStepwise: [-C, -B, -N, 98]
2.86 (20.00 / 7): weka.classifiers.functions.Logistic:[-R, 8.92512734844605E-5], null: [], NONE: []
2.91 (29.09 / 10): weka.classifiers.meta.Vote:[-R, AVG, -S, 1, -B, weka.classifiers.lazy.KStar -B 6 -M n], weka.attributeSelection.CfsSubsetEval: [-L], weka.attributeSelection.GreedyStepwise: [-C, -B, -R]
2.91 (29.09 / 10): weka.classifiers.lazy.LWL:[-A, weka.core.neighboursearch.LinearNNSearch, -W, weka.classifiers.lazy.IBk, --, -K, 1], null: [], NONE: []
2.91 (29.09 / 10): weka.classifiers.functions.Logistic:[-R, 0.0020589705876213438], weka.attributeSelection.CfsSubsetEval: [-M, -L], weka.attributeSelection.GreedyStepwise: [-C, -N, 53]
2.91 (29.09 / 10): weka.classifiers.lazy.LWL:[-K, 120, -A, weka.core.neighboursearch.LinearNNSearch, -W, weka.classifiers.functions.SMO, --, -C, 0.865139219625099, -N, 0, -M, -K, weka.classifiers.functions.supportVector.Puk -S 9.31797731197373 -O 0.3068500462454044], weka.attributeSelection.CfsSubsetEval: [-M, -L], weka.attributeSelection.BestFirst: [-D, 2, -N, 6]
2.91 (29.09 / 10): weka.classifiers.bayes.NaiveBayes:[-K], weka.attributeSelection.CfsSubsetEval: [-M], weka.attributeSelection.GreedyStepwise: [-C, -N, 15]
2.91 (29.09 / 10): weka.classifiers.lazy.KStar:[-B, 68, -M, n], weka.attributeSelection.CfsSubsetEval: [], weka.attributeSelection.BestFirst: [-D, 1, -N, 4]
2.91 (29.09 / 10): weka.classifiers.bayes.NaiveBayes:[-K], weka.attributeSelection.CfsSubsetEval: [-M, -L], weka.attributeSelection.GreedyStepwise: [-C, -N, 382]
3.03 (9.09 / 3): weka.classifiers.rules.DecisionTable:[-E, acc, -S, weka.attributeSelection.BestFirst, -X, 1], weka.attributeSelection.CfsSubsetEval: [-L], weka.attributeSelection.GreedyStepwise: [-B, -N, 125]
3.03 (9.09 / 3): weka.classifiers.meta.RandomSubSpace:[-I, 52, -P, 0.3957305480960892, -S, 1, -W, weka.classifiers.rules.OneR, --, -B, 1], null: [], NONE: []
3.90 (27.27 / 7): weka.classifiers.meta.AdaBoostM1:[-P, 89, -I, 24, -S, 1, -W, weka.classifiers.rules.PART, --, -M, 25], null: [], NONE: []
3.91 (39.09 / 10): weka.classifiers.functions.MultilayerPerceptron:[-L, 0.3813523419289583, -M, 0.8870112088084634, -B, -H, t, -S, 1], weka.attributeSelection.CfsSubsetEval: [-M], weka.attributeSelection.GreedyStepwise: [-C, -B, -R]
4.03 (28.18 / 7): weka.classifiers.trees.RandomForest:[-I, 6, -K, 3, -depth, 0], null: [], NONE: []
4.16 (29.09 / 7): weka.classifiers.functions.Logistic:[-R, 4.2207013384891565E-4], null: [], NONE: []
4.82 (48.18 / 10): weka.classifiers.functions.Logistic:[-R, 2.448575719895999E-8], weka.attributeSelection.CfsSubsetEval: [], weka.attributeSelection.GreedyStepwise: [-C, -R]
4.91 (49.09 / 10): weka.classifiers.meta.AdaBoostM1:[-P, 100, -I, 2, -S, 1, -W, weka.classifiers.functions.Logistic, --, -R, 2.628587683174939E-4], null: [], NONE: []
// I've truncated the list for the pull-request

------- 5 BEST CONFIGURATIONS -------

These are the 5 best configurations, as ranked by SMAC
Please note that this list only contains configurations evaluated on every fold.
If you need more configurations, consider running Auto-WEKA for a longer time.

Configuration #1:
SMAC Score: 2.909090909090909
Argument String:
-_0__wekaclassifierslazylwl_00_K_HIDDEN 1 -_0__wekaclassifierslazylwl_02_U_HIDDEN 1 -_0__wekaclassifierslazylwl_04_A weka.core.neighboursearch.LinearNNSearch -_1_W weka.classifiers.lazy.IBk -_1_W_0_DASHDASH REMOVED -_1_W_1__wekaclassifierslazyibk_00_E REMOVE_PREV -_1_W_1__wekaclassifierslazyibk_01_INT_K 1 -_1_W_1__wekaclassifierslazyibk_02_X REMOVE_PREV -_1_W_1__wekaclassifierslazyibk_03_F REMOVE_PREV -_1_W_1__wekaclassifierslazyibk_04_I REMOVE_PREV -attributesearch NONE -attributetime 10.0 -targetclass weka.classifiers.lazy.LWL 

Configuration #2:
SMAC Score: 2.909090909090909
Argument String:
-_0__wekaclassifierslazylwl_00_K_HIDDEN 1 -_0__wekaclassifierslazylwl_01_1_K 120 -_0__wekaclassifierslazylwl_02_U_HIDDEN 0 -_0__wekaclassifierslazylwl_04_A weka.core.neighboursearch.LinearNNSearch -_1_W weka.classifiers.functions.SMO -_1_W_0_DASHDASH REMOVED -_1_W_1__wekaclassifiersfunctionssmo_00_0_C 0.865139219625099 -_1_W_1__wekaclassifiersfunctionssmo_01_1_N 0 -_1_W_1__wekaclassifiersfunctionssmo_02_2_M REMOVED -_1_W_1__wekaclassifiersfunctionssmo_03_3_REG_IGNORE_QUOTE_START_K weka.classifiers.functions.supportVector.Puk -_1_W_1__wekaclassifiersfunctionssmo_08_4_puk_S 9.31797731197373 -_1_W_1__wekaclassifiersfunctionssmo_09_4_puk_O 0.3068500462454044 -_1_W_1__wekaclassifiersfunctionssmo_11_5_QUOTE_END REMOVED -aseval__wekaattributeselectioncfssubseteval_00_0_M REMOVED -aseval__wekaattributeselectioncfssubseteval_01_1_L REMOVED -assearch__wekaattributeselectionbestfirst_00_0_D 2 -assearch__wekaattributeselectionbestfirst_01_1_INT_N 6 -attributeeval weka.attributeSelection.CfsSubsetEval -attributesearch weka.attributeSelection.BestFirst -attributetime 10.0 -targetclass weka.classifiers.lazy.LWL 

[larskotthoff]

Thanks! This information is available already in the Auto-WEKA logs though in the temporary directory it creates (although not in such a nice format).

I'll consider this for inclusion in the package, but could you add an option to turn this on and off in the UI please? Also, we aim for Java 1.6 compatibility (and this is why our automated build system Travis failed).

[larskotthoff]

Thanks, I've had a closer look at this now. Could you please add something in the tooltip or help to explain what the numbers in parentheses are? Also, you need to filter the special dummy value (INFINITY in ClassifierResult).

[phiferd]

ok. I've been playing around with it on a few datasets and I've made a few minor improvements to my changes. I'll update it to include your suggests and update the PR when it's ready.

On Wed, Sep 7, 2016 at 1:59 AM, Lars Kotthoff [email protected] wrote:

Thanks, I've had a closer look at this now. Could you please add something in the tooltip or help to explain what the numbers in parentheses are? Also, you need to filter the special dummy value (INFINITY in ClassifierResult).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/automl/autoweka/pull/26#issuecomment-245035384, or mute the thread https://github.com/notifications/unsubscribe-auth/AABkH1WfuaSg52wTx-xk7tFRQkPZMTcKks5qnaoJgaJpZM4JxhxZ .

[larskotthoff]

Great, thanks!

[rleplae]

Would it be possible to see the proposed java code (to include in your own project) for more than just the best configuration ? Now i get an argument string, but that is kind of hard to decode what it exactly stands for...

[larskotthoff]

@rleplae We don't have any plans for this at the moment, but pull requests are welcome.

[rleplae]

I was able to download the AutoWeka source. took me a bit of time to figure out how to build, but i managed to import a new package

My notes :

• install Java jdk
• install Ant
• install Latex tool

Build the package

ant weka-package