Roman Lutz

Just to make sure I understand: You want to just run the example in a test case for the tensorflow-specific CI, right? I had never thought about this, but this...

Hmm, I have actually seen these tests catch a few issues that we could avoid since they were triggered on PRs. I'm not a huge fan of cronjobs for such...

Taking a step back: is that a useful page to have? [I created it so I don't feel bad questioning this 🤣 ]

I can reproduce this using this minimal example: ``` _create_group_metric_set(y_true=[0,1], predictions={'model1': [1,0]}, sensitive_features={"a": [0, 1], "b": [1, 3]}, prediction_type='binary_classification') ``` However, it does not happen with ``` _create_group_metric_set(y_true=[0,1,1,0], predictions={'model1': [1,0,1,0]},...

Good to hear that @ameyn21! @riedgar-ms evidently people are unsure what to do with these errors so we should at least surface which metric failed to process the input data....

The fact that I can't run this function when one group only has 1s or only 0s is sufficient proof that we're not handling error cases that definitely should be...

By dumping I meant exporting as a pre-step to serialization. Just as a dict basically. Maybe we have that already on the MetricFrame and I'm not aware of it...

We may want to reevaluate the level of detail and math, too. Perhaps yours is more of a deep dive than a "user guide" @riedgar-ms ? I feel like there...

Hi @IanEisenberg! Do you have a particular application scenario where this metric may be useful? I'm just curious where this may be helpful (vs. for example, equalized odds).

Good points! I'm not opposed to it but we really should put some effort into discussing pros and cons of these metrics. We should open an issue for that since...