BLIP2_Vicuna zero-shot performance on the AOKVQA dataset

[qwqwq1445]

trafficstars

I try to validate BLIP2_vicuna7b on the AOKVQA dataset. With the prompt "Question: {} Answer:", I got an accuracy of 12.34. Is it a reasonable number? I wonder if anyone also gave it a try.

[qwqwq1445]

I found this in the InstructBLIP paper and the paper shows an accuracy around 60.0. Unbelievable! The numerical difference is so large! I would appreciate it if arthors could tell me the exact hyperparameters for Vicuna-based BLIP2's zero-shot inference of AOKVQA! Thanks very much!

[Shwai-He]

I met similar issues, may I ask the verson of your used vicuna?

[Hurwitzzz]

I try to validate BLIP2_vicuna7b on the AOKVQA dataset. With the prompt "Question: {} Answer:", I got an accuracy of 12.34. Is it a reasonable number? I wonder if anyone also gave it a try.

I used BLIP2 opt2.7 and t5xl, and got a similar zero-shot acc on AOKVQA as yours and also have no idea.