MR-Models

MR-Models copied to clipboard

We have no choice: MultipleChoiceTask a few TMMLU Multiple Choice have no choice.

2

I have encountered with the evaluation datasets on TMMLU. Specifically, in the [分科測驗化學](https://github.com/mtkresearch/MR-Models/blob/main/TC-Eval/data/TMMLU/subjects/%E5%88%86%E7%A7%91%E6%B8%AC%E9%A9%97%E5%8C%96%E5%AD%B8/data.csv) and many more topics dataset, I observed that only question 1 includes choices in the query, whereas...

ZoneTwelve

您好，想請教此模型是用什麼FM進行fine-tune的呢?

gammaking

I found there are 3 subjects: 1. TMMLU/計算機數學 2. TMMLU/分科測驗數學甲 3. TMMLU/分科測驗化學 listed in the config.json file but missed in the data folder. Are the file removed? If so, I...

chc-tw

Fixes #18. Update `TTQA_mc_2.0.0.json` in `aggregate_results.py` and `scenarios.py` to `TTQA_mc.json`.

pha123661

In the directory TC-Eval/data/TTQA/, there is a file named **TTQA_mc.json**, but the following two lines of code incorrectly refer to the file as **TTQA_mc_2.0.0.json**. [scenarios.py](https://github.com/mtkresearch/MR-Models/blob/84e91b8bcf8d251483860ed55b9c78c76dbf772b/TC-Eval/inference/scenarios.py#L58) [aggregate_results.py](https://github.com/mtkresearch/MR-Models/blob/84e91b8bcf8d251483860ed55b9c78c76dbf772b/TC-Eval/inference/aggregate_results.py#L33)

teds-lin

For example: Index [54](https://github.com/mtkresearch/MR-Models/blob/84e91b8bcf8d251483860ed55b9c78c76dbf772b/TC-Eval/data/TMMLU/subjects/%E8%B3%87%E8%A8%8A%E5%AE%89%E5%85%A8/data.csv#L56) `54,C,"7. 「許多惡意程式會透過郵件附件方式進行入侵破壞與竊取，造成企業 重大損失，而資安工程師會透過 Windows 群組原則物件（Group Policy Object, GPO）進行必要管控。」關於附圖登錄檔（Registry）範例內容， 下列敘述何者正確？ (A) 放行 WordPad 開啟附件 (B) 支援 Office 2003 GPO 控制 (C) 對 Office 2016 MS-Word 禁用巨集（Macro） (D)...

teds-lin