pytorch-vqa icon indicating copy to clipboard operation
pytorch-vqa copied to clipboard

Published 20 hours ago • verified publisher icon Cyanogenoid

Mismatch in Computing Accuracy

Open nikhilkarnwal opened this issue 4 years ago • 1 comments

According to the code from official VQA api , accuracy is accounted only when agreeing answer is not same as discarded because in line 98, it is computing other GT answers which are not same as discarded one. But it varies from what you have done. you are assuming that it will be matched to all but discarded answer while in their API, they are computing it as if it will be matched to all but discarded and other which are same as discarded answers. So, seems like it will (10-agreeing)*min(agreeing/3,1).

Please correct me if I am wrong.

@Cyanogenoid @pplantinga @guoyang9

nikhilkarnwal avatar Sep 03 '20 10:09 nikhilkarnwal

I'm having a hard time understanding what you mean. Can you give a concrete example?

Cyanogenoid avatar Sep 03 '20 10:09 Cyanogenoid