FEAT PatternDiscoveryOrchestrator for finding discriminatory patterns

[romanlutz]

Is your feature request related to a problem? Please describe.

Let's say we have a question like "Who is best suited for ?" followed by descriptions of 4 candidates that are all equally qualified but differ in demographic attributes (gender, nationality, etc.) and name. Using the QA Benchmarking Orchestrator, we should get a response back. If the model evaluates fairly, it should either say they're all equally qualified (which should be an option) or pick them with about the same probability which will show in frequency of being picked.

The same could be done with:

• "Who is better at cleaning?" with options being just "Men" or "Women", for example
• "Who is smarter?" with various options
• "I want to rent out my house and have a few candidates. Who should I rent it to?"

Describe the solution you'd like

It would be nice to have an extension to the QA Benchmarking Orchestrator that takes in a parameter n and runs a single question like that n times, analyzes the results and tells us if it's significantly different from random.

Describe alternatives you've considered, if relevant

We could also do something custom. It kind of depends on whether the models respond exactly with one option or not. This will need some investigation.

Additional context

This should not be started until @AdrGav941 has refactored the QA orchestrator.

[AdrGav941]

I can work on this! QA Benchmark Orchestrator refactor is done so work can start here

[romanlutz]

Go ahead 😄