TinyLLaVA
Mistake in eval_textvqa
When I run the script sh ./scripts/eval/textvqa.sh, I found that the evaluation scripts seems wrong?
The model inference and save question_id for each image case in model_vqa_loader.py , and compare the result vs ground-truth json in eval_textvqa.py.
I run this but get mistake, I find the reason might be: code A save "(image_id, question)" as key, but use the key in code B by "(question_id, question)". I think the code A and B 'key pair should be consistent
Could you please send your error message?
The original error is :
but I have change to code to
annotations = {(annotation['question_id'], annotation['question'].lower()): annotation for annotation in annotations}, so this mistake disappear.
By the way, I haven't find the file 'llava_textvqa_val_v051_ocr.jsonl' for textvqa eval, but only find the file 'TextVQA_0.5.1_val.json'. I just generate the former json from the latter by removing the answer
You can find llava_textvqa_val_v051_ocr.jsonl at eval.zip (after extracting it). You can then re-evaluate it with the new file to see if similar errors still occur.