armory
armory copied to clipboard
WIP per-class metrics
For #492
@davidslater The poisoning scenario adds this metric automatically, right in the code. Is this an acceptable way to make it available to other scenarios through the config/metric/task section?
I like adding it to the config part of instrumentation.
@davidslater So far I've added (beside per-class accuracy) a confusion matrix, and precision and recall for each class. Two questions: Will you check that I'm computing the right thing (I described my understanding in the comments), and that the output is in a sufficiently useful format (dict, array, etc)? Can you think of any other metrics or statistics that would be nice to have?
See my two recent comments. Beyond that, I think what needs to be done is:
A) remove the changes to armory/instrument/config.py
, and armory/utils/config_schema.json
.
B) move the new functions from statistical
to task
, since we're registering them as task populationwise metrics anyhow.
C) Probably easiest to rebase off of develop.