Vision-SR1 Some test datasets are unavailable

Thanks for your excellent work. When I was evaluating the self_reward_7b model, I found that some datasets seemed to be invisible on huggingface, such as " zli12321/mm-vet", "zli12321/pope" . How can I obtain the corresponding data set to reproduce the results?

Sep 22 '25 09:09 kkkkkkon

They should be available now Pope MM-Vet

Sep 23 '25 03:09 zli12321

Thank you very much. By the way, have you considered to release the Vision-SR1-3B model?

They should be available now Pope MM-Vet

Sep 23 '25 04:09 kkkkkkon

It's here. 3B is less popular though.

Sep 23 '25 04:09 zli12321

We evaluated the downloaded results and found they didn't match the results in the paper. After checking, we found that the Vision-SR1 test results and Vision-SR1-ablation were misplaced. For example, the MMMU in Vision-SR1-ablation corresponded to the results in the paper, while the MMMU in Vision-SR1 corresponded to the ablation results.

Sep 24 '25 03:09 kkkkkkon

thank you for pointing the issue out. Indeed there are some misplaced paths. Rerunning the MCQ now to make sure they are put under the right path.

If it is urgent, pls email us.

Sep 24 '25 04:09 zli12321

We just confirmed MMMU acc is 55.3. Thanks for pointing out. Will update all the results accordingly.

Sep 24 '25 05:09 zli12321