Haodong Duan

Hi, @juxingyiwan , custom multiple-choice dataset is supported now. Just prepare your dataset following the typical format of MCQ datasets in VLMEvalKit, place it under ~/LMUData, and use the tsv...

Hi, @John-Ge , 1. For simplicity reasons, VLMEvalKit do not support batch size > 1 inference for now. 2. VLMEvalKit currently supports two types of multi-GPU inference: 1). DistributedDataParallel via...

Hi, @darkpromise98 , we will try to include this feature into VLMEvalKit recently.

Hi, @John-Ge @darkpromise98 , I have reviewed the request. I'm sorry that I may not implement this feature on my own for the following reasons: 1. Currently, only few VLMs...

Sorry, the project was supported by an internal version of caffe, so the code may not be available.

Sorry that the code for this project is not available at present, since most of our experiments were built on top of internal codebase. We will release the code after...

Sorry for the late reply, currently we do not have plan for releasing COCO-trb annotations.

Would you please further clarify the problem, It's OK to use Chinese.

Oh, we think it's OK to use MMMU_DEV_VAL, which only incurs an overhead of 150 visual questions. Besides, in the accuracy file we provide the scores for both MMMU_DEV and...