zero-shot accuracy of instructblip on ScienceQA

[hongliang-wei]

Thank you for your outstanding work.

1. I noticed that the paper mentions "For ScienceQA, we only evaluate the set with image context." Does this mean that the hint or context in the dataset is not used?
2. What is the prompt template used by instructblip_flant5xl on ScienceQA?
3. I use "Question: {}\n{}" as prompt and the zero-shot accuracy of instruct_flant5xl on ScienceQA i obtained is 64.6。Are there any things I need to pay attention to?

[wenliangdai]

Thanks for your interests in our work!

1. It means we only use the questions with the image context (the IMG set). We do include the textual context if exists.
2. and 3. "Context: {} Question: {} Options: {}. Answer:", options are separated by (a), (b), (c), (d). In addition, we use answer ranking for interence as mentioned in Section 2.5

[hongliang-wei]

Thanks for the quick response.

In ScienceQA, some questions have options with images. Does InstructBlip utilize these images?

[engineerA314]

What did you use as Context in ScienceQA? I used "hint" in scineceQA but I got only 47.8% accuracy.

And Did you use predict_class in ScienceQATask's valid_step?