We have no choice: MultipleChoiceTask a few TMMLU Multiple Choice have no choice.

[ZoneTwelve]

I have encountered with the evaluation datasets on TMMLU.

Specifically, in the 分科測驗化學 and many more topics dataset, I observed that only question 1 includes choices in the query, whereas the ground truth provides corresponding answers.

I believe addressing this discrepancy is crucial to maintaining the integrity and accuracy of the TMMLU evaluation process. Your prompt attention to this matter would be highly appreciated.

Here's the topic with the same issues:

• 分科測驗數學甲
• 計算機數學
• 分科測驗物理

---

How and Why does that cause an issue? Because the query prompt doesn't include the choices.

[ZoneTwelve]

Please review this example if you don't believe there is an issue. TMMLU/分科測驗數學甲:3 TMMLU/分科測驗數學甲:4

References	Prompt
B	<s> [INST] <<SYS>> <</SYS>> 15-17題為題組坐標空間中，有一立體Γ的底面位於xy平面上，且底面是由兩拋物線y=x2與 y=8-x2圍成的區域，而Γ的每一個垂直x軸的截面都是等腰直角三角形，且三角形的斜邊在xy平面上。設每一個垂直x軸且通過2(t ,t2,0)的截面所成等腰直角三角形面積為 A(t) 。試回答下列問題。【題組】15. 試求此立體Γ的底面面積。 (4 分 ) [/INST]
D	<s> [INST] <<SYS>> <</SYS>> 【題組】16.試以t表示A(t)。(2分) [/INST]

[Splend1d]

Thank you. We are working on solving this matter.