armory icon indicating copy to clipboard operation
armory copied to clipboard

WIP per-class metrics

Open swsuggs opened this issue 2 years ago • 4 comments

For #492

swsuggs avatar May 27 '22 17:05 swsuggs

@davidslater The poisoning scenario adds this metric automatically, right in the code. Is this an acceptable way to make it available to other scenarios through the config/metric/task section?

swsuggs avatar May 27 '22 17:05 swsuggs

I like adding it to the config part of instrumentation.

davidslater avatar May 27 '22 19:05 davidslater

@davidslater So far I've added (beside per-class accuracy) a confusion matrix, and precision and recall for each class. Two questions: Will you check that I'm computing the right thing (I described my understanding in the comments), and that the output is in a sufficiently useful format (dict, array, etc)? Can you think of any other metrics or statistics that would be nice to have?

swsuggs avatar May 31 '22 17:05 swsuggs

See my two recent comments. Beyond that, I think what needs to be done is: A) remove the changes to armory/instrument/config.py, and armory/utils/config_schema.json. B) move the new functions from statistical to task, since we're registering them as task populationwise metrics anyhow. C) Probably easiest to rebase off of develop.

davidslater avatar Dec 06 '22 00:12 davidslater