VLMEvalKit
VLMEvalKit copied to clipboard
fix(image_mcp.py): use consistant argument (#929)
fix argument error in MMMU_Pro benchmark, close #929
Hi, @MaoSong2022 ,
It looks like every other dataset do not set the arg names explicitly, so I don't think the modification is necessary.
the change is to keep the arguments consistent with ImageBaseDataset initialization. For now, only a few dataset implements this method.
ImageBaseDatasetinitialization:def __init__(self, dataset='MMBench', skip_noimg=True)- some other dataset initialization (such as
MMGenBench):def __init__(self, dataset, **kwargs) - this commit (related issue):
def __init__(self, dataset='MMMU_Pro', skip_noimg=True)
the question is should we use consistent arguments when inheriting ImageBaseDataset, if the answer is yes, which format should we use.