Reproducibility Issues (Bias Score and Accuracy)

[rajneesh407]

I have been working on reproducing the results using the code from your repository and implementing it in Python. I successfully converted the provided R code into Python, and the outputs from both versions match. However, the results from your repository code do not align with the figures presented in your research paper.

Code used: [BBQ_calculate_bias_score.R] (https://github.com/nyu-mll/BBQ/blob/main/analysis_scripts/BBQ_calculate_bias_score.R) Research paper link: [QA Bias Benchmark] (https://github.com/nyu-mll/BBQ/blob/main/QA_bias_benchmark.pdf)

Here is the output I obtained using the R code from your repository:

Comparing Dberta V3 Base (For Disambiguous): Comparison With Paper : - Age : Match - Disability : Match - Gender Identity : 13.9 (R-Code) instead of 15 (paper) - Gender Identity Names : 12.7 instead of 14 - Nationality : Match - Physical Appearance : 41.8 instead of 41 - Race and Ethnicity : 4.7 instead of 4.6 - Race and Ethnicity Names : Match - Religion : Match - Sexual Orientation : Match - SES : Match

Same patter can be seen across other models also.

Any help/clarifications would be appreciated here. Thanks