Adding a list of the instruction prompts used during training

[NohTow]

Hello,

I am trying to test the outputs of OFA (by hand) and I find it a bit difficult to be 100% sure of how to prompt the model to remain consistent with the training. I know the prompts are written within the paper, but given how sensible the model can be to different prompts, I am concerned about spaces/cases. For example, the prompt for captioning given in the paper is "[Image] What does the image describe?" whereas, the prompt given as example in the code is " what does the image describe?". The space in the beginning is surely there to make the separation between the tokens of the image and the ones from the text, but what about the lowercase ? Also, should I put quotes around the text modality when it is within the prompt ?

Given how many issues there is about how to adapt the model to different dataset, I think it would be really helpful to have the complet list of tasks/associate prompt (in string format), like the Table 12 of the paper, but directly in the documentation and with the exact string. Besides, adding it to the HF space could help people to try the model without having to dig too much.

I am sure these information are foundable within the code, but it would help people that are starting with the model.

[JustinLin610]

Thanks for pointing out the problem. Yep this might be confusing for people paying attention to details. Actually in our code we all use lowercase, but when we write the paper, we did not pay attention to this issue carefully and write as we usually do. Let me confirm your suggestion: can I create a txt file that lists tasks and their corresponding prompts?

[NohTow]

Yeah, I understand that this kind of detail is not what you have in mind when writting the paper, that's why I made this proposition. I'm not really too worried about details, but since you mentionned in the paper that the model is very sensible to the prompt, I want to make sure I do not have worse results because of that.

Either a txt file, or a table in the readme (basically the same as Table 12 but with real prompts) would be awesome.