Failed to download MMBench_V11.tsv error

[OfirArviv]

Hi, I'm trying to run the evaluation on the published records (https://huggingface.co/datasets/VLMEval/OpenVLMRecords), and for the MMBench file (for example Claude3-5V_Sonnet_MMBench_V11.xlsx) I receive the following error?

Exception: Failed to download https://opencompass.openxlab.space/utils/benchmarks/MMBench/MMBench_V11.tsv

I'm running vlmutil eval "Claude3-5V_Sonnet_MMBench_V11.xlsx" for example.

Thanks!

[PhoenixZ810]

Hi,

Thank you for your interest in VLMEvalKit!

Please note that MMBench_V11.tsv is not publicly available. If you'd like to evaluate MMBenchV11_TEST_EN, you can use the following command:

python run.py --model xx --data MMBench_TEST_EN_V11

After generating the results, please submit them to the MMBenchOnline platform to obtain the evaluation score.

If you have any further questions or need additional assistance, feel free to reach out!

Best regards,

[OfirArviv]

Thanks for the reply. I already have the predictions from your evaluation records data. I'm only looking to evaluate the response.

Per your instruction. I"m running

vlmutil eval "Claude3-5V_Sonnet_MMBench_V11.xlsx" for example. There are also MMBench.xlsx files in the records. But they don't work as well