Error when using MetricFrame with sample_params

[LengMei]

Hi, I am using MetricFrame to generate evaluations with a set of built-in metrics with extra parameters. The parameters are passed to MetricFrame using sample_params. I found that for built-in metrics with sample_weight and pos_label as extra parameters, sample_weight is always working well, but when pos_label was passed with a not-None value, it produced IndexError from fairlearn/metrics/_function_container.py. I've tried all *_rate metrics imported from fairlearn.metrics, like false_positive_rate, and also similar metrics imported from sklearn.metrics, like recall_score. All behaved similarly.

Here is the minimal code to reproduce, and I am using fairlearn 0.7.0

import fairlearn.metrics as flm
from fairlearn.metrics import MetricFrame
y_true = [0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1]
y_pred = [0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1]
gp = ['d', 'a', 'c', 'b', 'b', 'c', 'c', 'c',
                         'b', 'd', 'c', 'a', 'b', 'd', 'c', 'c']
sample_weight = [0.1]*len(y_true) 
pos_label = 1
print(flm.false_positive_rate(y_true, y_pred, sample_weight, pos_label))
gm = MetricFrame(metrics=flm.false_positive_rate,
                             y_true=y_true,
                             y_pred=y_pred,
                             sensitive_features=gp,
                sample_params={'sample_weight': sample_weight, 'pos_label': pos_label})
print(gm.overall)

The above code will print 0.666666 for the direct computation by calling the metric function, and throw error when using in MetricFrame. The last part of the error message is:

/home/cdsw/.local/lib/python3.6/site-packages/fairlearn/metrics/_function_container.py in generate_sample_params_for_mask(self, mask)
     82         curr_sample_params = dict()
     83         for name, value in self.sample_params_.items():
---> 84             curr_sample_params[name] = value[mask]
     85         return curr_sample_params
     86 

IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed

and the screenshot is:

When setting pos_label to None, both computation will work and print out the same result.

Thanks in advance for any suggestion and help.

[LeJit]

Hi LengMei,

Thanks for opening this issue and finding this bug! sample_params is expecting array-like parameters with length matching that of y_true and y_pred. Unfortunately, passing in pos_label as an array (e.g. "pos_label" = [pos_label]*len(y_true)) won't fix the problem since the false_positive_rate function is expecting pos_label to be a scalar value.

It should be a simple fix on our end to update the MetricFrame to accept scalar and array-like values for sample_params.

[romanlutz]

@LeJit are you working on this or should we unassign it so that someone else can take it up?