visdial-rl icon indicating copy to clipboard operation
visdial-rl copied to clipboard

Published 20 hours ago • verified publisher icon batra-mlp-lab

Fix bug in evaluation for tie in scores.

Open kdexd opened this issue 6 years ago • 3 comments

If the score of ground truth would tie with another option, it always got assigned a higher rank. This caused evaluation metrics to be reported higher than their true values. This PR fixes that.

kdexd avatar Jul 25 '18 20:07 kdexd

So this change would require re-evaluating all checkpoints and updating the reported numbers in the README, right?

nirbhayjm avatar Jul 26 '18 04:07 nirbhayjm

I just reviewed https://github.com/batra-mlp-lab/visdial/pull/36 and tried simplifying things a bit: https://github.com/batra-mlp-lab/visdial/pull/36/commits/3a769d28d15c5112fccaefdc041572d069f302f2. Those changes probably apply here too.

abhshkdz avatar Jul 26 '18 06:07 abhshkdz

@nirbhayjm yes it does. If we never faced ties in scores, then they won't change. But usually there are ties...

kdexd avatar Jul 26 '18 14:07 kdexd