Addressing the Differing Data Distributions in the GAP Test Set

[OanaMariaCamburu]

As the authors mentioned, the instances in GAP have certain differences between the female and male groups, e.g., the correct candidate is on average further away from the pronoun in the female group than in the male group, the female instances also contains more candidates on average than the male instances. These would incorrectly make unbiased models appear biased.

In our work, we address this issue. Check it out if you are using the GAP test set as a gender bias diagnostic dataset.

@misc{kocijan2020gap,
      title={The Gap on GAP: Tackling the Problem of Differing Data Distributions in Bias-Measuring Datasets}, 
      author={Vid Kocijan and Oana-Maria Camburu and Thomas Lukasiewicz},
      year={2020},
      eprint={2011.01837},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
 

[sandeep16064]

right