Is there a bug in mvbench check_ans?

[sunwhw]

trafficstars

I noticed that sometimes the model would return an empty answer"". However, the empty answer is turned out to be correct by the "check_ans".

[Andy1621]

Good question! Can you provide some cases to generate an empty answer? In my experiments, the model always generates options with the answer prompt.

[sunwhw]

oh, I am testing some unpublic model, it give empty response sometimes, I think it is the mvbench prompt doesn't fit the model, so it give such a strange response.

[Andy1621]

You can change the evaluation code to fit your model~

[sunwhw]

ok, Thanks!