sjwhitworth
Many tests don't make assertions
A large number of tests don't make any assertions, and instead just print stuff out, e.g. a confusion matrix summary. Seems to defeat the purpose of having automated tests. There is some value to these tests, if they pass, it tells you the algorithm runs end to end without blowing up. But I've noticed when running in verbose mode that some of the algorithms produce 0% accuracy, and when tweaking parameters to improve the accuracy, the tests crashed with a memory panic.
Seems like having actual assertions (in addition to being valuable on their own) would've helped catch that kind of thing earlier. I'm willing to volunteer to go do this, work through the tests and make sure they're all making valuable assertions, someone just needs to assign it to me.
While I'm at it, any ideas for what sorts of things would be valuable to assert on? At minimum, for things that were printing out confusion matrices, asserting on the overall summary is a start.
0% accuracy? That sounds pretty unusual.
I think you need to be a collaborator to get assigned to this issue. You've had a pull request merged, so I'm sure @sjwhitworth can sort that out.
Yup. 0% for one of the three iris labels, not 0% over all. And it wouldn't happen all the time, but about 1 in 4 runs of the test. Increasing the size of the RF helped things, but then I started seeing the memory panics which you know about and fixed.
So one way I found of correcting the problem is to reduce the significance level for discretisation to 0.6 and also reduce the amount of data used for pruning. This helps the ID3 algorithm to build a better tree and so seems to reduce the variability of the results (at least on my machine). Relevant branch.