deepchecks

deepchecks copied to clipboard

Background

In order to provide extra value in performance & drift checks, we want to always display information about these quantities in an aggregate manner (as done in the reduce function of drift, or with aggregate metrics in performance checks) and also in a per class or per feature manner.

DoD

1. In drift checks, the reduce output will always be both the aggregate value and the value per feature, in a format defined by @JKL98ISR
2. In performance checks, if a check receives a known string for an aggregate metric (such as f1_macro), the check will add to the metric list also the per class version (in this case "f1_per_class"). This extra metric will appear in the display as if it was added by the user, and will also be included in the reduce output in a format defined by @JKL98ISR .

Tasks:

1. Define a dict / other structure connecting each built-in aggregate metric to it's per-class version. 0.5 sp
2. Make sure each performance check adds the per-class metric if given an aggregate one. Document that behavior in the check docs and plot. 1.5 sp
3. Change the reduce output of the performance and drift checks to the format defined by @JKL98ISR. 1 sp
4. Add tests for the new behavior of the reduce output 0.5 sp