MMLU Template Instruction Inconsistency

[RonaldBXu]

Hi, I noticed that in some templates of MMLU, it seems that the part of the prompt "The following are multiple choice questions (with answers) about {topic}." is being placed in "input_format", when it should be in "instruction", right? This is because we don't want to repeat this at the beginning of each shot?

This is in unitxt/prepare/cards/mmlu.py and unitxt/prepare/templates/qa/multiple_choice/templates.py.

[yoavkatz]

You are right, In the past, unitxt did not support the "instruction" field and these are older templates. There are templates that use the instruction field like "templates.qa.multiple_choice.with_context.with_topic.title"

BTW, it's sometimes not clear which is the better option. From some experiments on one model , we saw that repeating the instruction per demo actually improves performance.