Add benchmark script for mlpack_decision_tree

[ThyrixYang]

related to #886 in mlpack

[mlpack-jenkins]

Can one of the admins verify this patch?

[zoq]

Can you add a simple test for the mlpack dtc benchmark script? Take a look at: https://github.com/zoq/benchmarks/blob/master/tests/benchmark_kernel_pca.py for how the test could look like.

[mlpack-jenkins]

Can one of the admins verify this patch?

[ThyrixYang]

Hi @zoq , I'm confused that I didn't find how these test script is used? I tried

python3 -m unittest benchmark_xxx.py

but not worked.

[zoq]

You can either run make checksor python3 tests/tests.py. Note that you need python-xmlrunner to run the tests.

[rcurtin]

@mlpack-jenkins test this

[ThyrixYang]

@rcurtin I can run the benchmark by "make run", and when I run the benchmark test locally:

thyrix@y400:~/Downloads/benchmarks$ make checks
/usr/bin/python3 tests/tests.py

Running tests...
----------------------------------------------------------------------
....EE
======================================================================
ERROR [0.001s]: test_Constructor (benchmark_decision_tree.DecisionTree_SHOGUN_TEST)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/thyrix/Downloads/benchmarks/tests/benchmark_decision_tree.py", line 62, in setUp
    module = Loader.ImportModuleFromPath("methods/shogun/decision_tree.py")
  File "/home/thyrix/Downloads/benchmarks/util/loader.py", line 46, in ImportModuleFromPath
    tup = imp.find_module(modName, ['.'])
  File "/usr/lib/python3.5/imp.py", line 296, in find_module
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'decision_tree'

======================================================================
ERROR [0.000s]: test_RunMetrics (benchmark_decision_tree.DecisionTree_SHOGUN_TEST)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/thyrix/Downloads/benchmarks/tests/benchmark_decision_tree.py", line 62, in setUp
    module = Loader.ImportModuleFromPath("methods/shogun/decision_tree.py")
  File "/home/thyrix/Downloads/benchmarks/util/loader.py", line 46, in ImportModuleFromPath
    tup = imp.find_module(modName, ['.'])
  File "/usr/lib/python3.5/imp.py", line 296, in find_module
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'decision_tree'

----------------------------------------------------------------------
Ran 6 tests in 0.499s

FAILED (errors=2)

Generating XML reports...
Makefile:237: recipe for target '.checks' failed
make: *** [.checks] Error 1

So I think mlpack_decision_tree benchmark passed the test. Can you help me with the failure of jenkins test? I have no idea about the error now.

[zoq]

The current jenkins setup uses mlpack 2.2.2 where you have to pass a seperate labels file: mlpack_decision_tree -t datasets/iris_train.csv [FATAL] Must specify --labels_file when --training_file is specified! this behavior was fixed in 2.2.3

So I think, what we can do here is to pass the labels file, and we should also update mlpack as soon as possible.

[ThyrixYang]

@zoq I see. I added that feature when I was writing this benchmark script. Because all datasets(train file) in benchmark system use the last column as labels. And all other methods go with it well. So I think it's better to merge this pr after mlpack is updated and it passes the tests.

[zoq]

Sounds reasonable, I'll update the mlpack package later today and rerun the test.

[rcurtin]

Hmm, I think we can't update the mlpack package until the currently running mlpack benchmarks finish, unfortunately. I suppose it could be a few days then.

[zoq]

If we update the package right now and run the checkout - all nodes job, it's not going to install the new version on the current machine that runs the mlpack benchmark until it finished right? Unless I missed something.

[rcurtin]

@mlpack-jenkins test this