VLMEvalKit
VLMEvalKit copied to clipboard
open-compass
[rank0]: KeyError: 'answer'
/vlmeval/dataset/image_mcq.py", line 181, in evaluate [rank0]: answer_map = {i: c for i, c in zip(meta['index'], meta['answer'])} [rank0]: File "/data/miniconda3/envs/env-3.9.2/lib/python3.9/site-packages/pandas/core/frame.py", line 4102, in getitem [rank0]: indexer = self.columns.get_loc(key) [rank0]: File "/data/miniconda3/envs/env-3.9.2/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc [rank0]: raise KeyError(key) from err [rank0]: KeyError: 'answer'
If you are evaluating a private benchmark, please check your dataset file xxx. tsv fields correct, such as 'answer' instead of 'ans'. Please refer to Development Doc for more detailed information.
I try to reproduce this error. This is my command torchrun --nproc-per-node=8 run.py --model InternVL-Chat-V1-5 --data MME. Everything is OK. And MME benchmark should correspond to the image_yorn.py file instead of image_mcq.py. Please check your code repo.
Then that's weired, I didn touched anything inside vlmkit, those tsv are automatically downloaded, there must be an key error at some where in some dataset.
Maybe you can try to pull our latest code. We have fixed some known minor bugs. If you still encounter other problems, please do not hesitate to contact us and give us some detailed information to fix them. Thank you.
Hi, @lucasjinreal , I tried to reproduce the problem but failed (I'm evaluating llava_v1.5_7b on MME), with latest main branch.
