InternVL icon indicating copy to clipboard operation
InternVL copied to clipboard
Published 3 months ago • verified publisher icon OpenGVLab

[Bug] Reproducing MMVet, AI2D Results

Open avdravid opened this issue 7 months ago • 2 comments

Checklist

  • [x] 1. I have searched related issues but cannot get the expected help.
  • [x] 2. The bug has not been fixed in the latest version.
  • [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

I am running VLMEvalkit on MMVet and AI2D with the following command:

export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 export USE_COT="1" torchrun --master_port=9989 --nproc-per-node=8 run.py --data MMVet AI2D_TEST --model InternVL3-38B

But I am not able to reproduce the result from either the paper or the VLMEvalkit leaderboard. I am getting 76.5 while VLMEvalkit reports 81.1 for MMVet. For AI2D, I get 84.5 vs the reported 88.7. I have tried running multiple times but the variance is pretty low. I am able to perfectly reproduce the results on MME and OCRBench, with a slightly deviated but acceptable performance on HallusionBench. Here is the environment info:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge accelerate 1.6.0 pypi_0 pypi addict 2.4.0 pypi_0 pypi altair 5.5.0 pypi_0 pypi annotated-types 0.7.0 pypi_0 pypi anyio 4.9.0 pypi_0 pypi aom 3.5.0 h27087fc_0 conda-forge argon2-cffi 23.1.0 pypi_0 pypi argon2-cffi-bindings 21.2.0 pypi_0 pypi arrow 1.3.0 pypi_0 pypi arxiv 2.2.0 pypi_0 pypi asttokens 3.0.0 pyhd8ed1ab_1 conda-forge async-lru 2.0.5 pypi_0 pypi attrs 25.3.0 pypi_0 pypi babel 2.17.0 pypi_0 pypi beautifulsoup4 4.13.4 pypi_0 pypi bitsandbytes 0.45.5 pypi_0 pypi blas 1.0 mkl
bleach 6.2.0 pypi_0 pypi blinker 1.9.0 pypi_0 pypi brotli-python 1.0.9 py310h6a678d5_9
bzip2 1.0.8 h4bc722e_7 conda-forge ca-certificates 2025.4.26 hbd8a1cb_0 conda-forge cachetools 5.5.2 pypi_0 pypi certifi 2025.4.26 pyhd8ed1ab_0 conda-forge cffi 1.17.1 pypi_0 pypi charset-normalizer 3.3.2 pyhd3eb1b0_0
click 8.2.0 pypi_0 pypi colorama 0.4.6 pypi_0 pypi comm 0.2.2 pyhd8ed1ab_1 conda-forge contourpy 1.3.2 pypi_0 pypi cuda-cudart 12.4.127 0 nvidia cuda-cupti 12.4.127 0 nvidia cuda-libraries 12.4.1 0 nvidia cuda-nvrtc 12.4.127 0 nvidia cuda-nvtx 12.4.127 0 nvidia cuda-opencl 12.9.19 0 nvidia cuda-runtime 12.4.1 0 nvidia cuda-version 12.9 3 nvidia cycler 0.12.1 pypi_0 pypi debugpy 1.8.14 py310hf71b8c6_0 conda-forge decorator 5.2.1 pyhd8ed1ab_0 conda-forge decord 0.6.0 pypi_0 pypi deepspeed 0.16.7 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi distro 1.9.0 pypi_0 pypi docstring-parser 0.16 pypi_0 pypi dotenv 0.9.9 pypi_0 pypi duckduckgo-search 5.3.1b1 pypi_0 pypi einops 0.8.1 pypi_0 pypi et-xmlfile 2.0.0 pypi_0 pypi exceptiongroup 1.3.0 pyhd8ed1ab_0 conda-forge executing 2.2.0 pyhd8ed1ab_0 conda-forge fastjsonschema 2.21.1 pypi_0 pypi feedparser 6.0.11 pypi_0 pypi ffmpeg 4.4.2 gpl_h8dda1f0_112 conda-forge filelock 3.17.0 py310h06a4308_0
flash-attn 2.7.0.post2 pypi_0 pypi font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_3 conda-forge fontconfig 2.15.0 h7e30c49_1 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.58.0 pypi_0 pypi fqdn 1.5.1 pypi_0 pypi freetype 2.13.3 ha770c72_1 conda-forge func-timeout 4.3.5 pypi_0 pypi giflib 5.2.2 h5eee18b_0
gmp 6.3.0 h6a678d5_0
gmpy2 2.2.1 py310h5eee18b_0
gnutls 3.7.9 hb077bed_0 conda-forge griffe 0.49.0 pypi_0 pypi h11 0.16.0 pypi_0 pypi h2 4.2.0 pypi_0 pypi hjson 3.1.0 pypi_0 pypi hpack 4.1.0 pypi_0 pypi httpcore 1.0.9 pypi_0 pypi httpx 0.28.1 pypi_0 pypi huggingface-hub 0.31.2 pypi_0 pypi hyperframe 6.1.0 pypi_0 pypi icu 75.1 he02047a_0 conda-forge idna 3.7 py310h06a4308_0
imageio 2.37.0 pypi_0 pypi importlib-metadata 8.6.1 pyha770c72_0 conda-forge intel-openmp 2021.4.0 h06a4308_3561
ipykernel 6.29.5 pyh3099207_0 conda-forge ipython 8.36.0 pyh907856f_0 conda-forge ipywidgets 8.1.7 pypi_0 pypi isoduration 20.11.0 pypi_0 pypi jedi 0.19.2 pyhd8ed1ab_1 conda-forge jinja2 3.1.6 py310h06a4308_0
jiter 0.9.0 pypi_0 pypi joblib 1.5.0 pypi_0 pypi jpeg 9e h5eee18b_3
json5 0.12.0 pypi_0 pypi jsonpointer 3.0.0 pypi_0 pypi jsonschema 4.23.0 pypi_0 pypi jsonschema-specifications 2025.4.1 pypi_0 pypi jupyter 1.1.1 pypi_0 pypi jupyter-console 6.6.3 pypi_0 pypi jupyter-events 0.12.0 pypi_0 pypi jupyter-lsp 2.2.5 pypi_0 pypi jupyter-server 2.16.0 pypi_0 pypi jupyter-server-terminals 0.5.3 pypi_0 pypi jupyter_client 8.6.3 pyhd8ed1ab_1 conda-forge jupyter_core 5.7.2 pyh31011fe_1 conda-forge jupyterlab 4.4.2 pypi_0 pypi jupyterlab-pygments 0.3.0 pypi_0 pypi jupyterlab-server 2.27.3 pypi_0 pypi jupyterlab-widgets 3.0.15 pypi_0 pypi keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.8 pypi_0 pypi krb5 1.21.3 h659f571_0 conda-forge lagent 0.2.4 pypi_0 pypi lame 3.100 h7b6447c_0
lazy-loader 0.4 pypi_0 pypi lcms2 2.15 hfd0df8a_0 conda-forge ld_impl_linux-64 2.43 h712a8e2_4 conda-forge lerc 4.0.0 h6a678d5_0
libcublas 12.4.5.8 0 nvidia libcufft 11.2.1.3 0 nvidia libcufile 1.14.0.30 4 nvidia libcurand 10.3.10.19 0 nvidia libcusolver 11.6.1.9 0 nvidia libcusparse 12.3.1.170 0 nvidia libdeflate 1.17 h0b41bf4_0 conda-forge libdrm 2.4.124 hb9d3cd8_0 conda-forge libedit 3.1.20250104 pl5321h7949ede_0 conda-forge libexpat 2.7.0 h5888daf_0 conda-forge libffi 3.4.6 h2dba641_1 conda-forge libfreetype 2.13.3 ha770c72_1 conda-forge libfreetype6 2.13.3 h48d6fc4_1 conda-forge libgcc 15.1.0 h767d61c_2 conda-forge libgcc-ng 15.1.0 h69a702a_2 conda-forge libgomp 15.1.0 h767d61c_2 conda-forge libiconv 1.18 h4ce23a2_1 conda-forge libidn2 2.3.4 h5eee18b_0
libjpeg-turbo 2.0.0 h9bf148f_0 pytorch liblzma 5.8.1 hb9d3cd8_1 conda-forge liblzma-devel 5.8.1 hb9d3cd8_1 conda-forge libnpp 12.2.5.30 0 nvidia libnsl 2.0.1 hd590300_0 conda-forge libnvfatbin 12.9.19 0 nvidia libnvjitlink 12.4.127 0 nvidia libnvjpeg 12.3.1.117 0 nvidia libpciaccess 0.18 hd590300_0 conda-forge libpng 1.6.47 h943b412_0 conda-forge libsodium 1.0.20 h4ab18f5_0 conda-forge libsqlite 3.49.2 hee588c1_0 conda-forge libstdcxx 15.1.0 h8f9b012_2 conda-forge libstdcxx-ng 15.1.0 h4852527_2 conda-forge libtasn1 4.19.0 h5eee18b_0
libtiff 4.5.0 h6adf6a1_2 conda-forge libunistring 0.9.10 h27cfd23_0
libuuid 2.38.1 h0b41bf4_0 conda-forge libva 2.18.0 h0b41bf4_0 conda-forge libvpx 1.11.0 h9c3ff4c_3 conda-forge libwebp 1.2.4 h1daa5a0_1 conda-forge libwebp-base 1.2.4 h166bdaf_0 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxcrypt 4.4.36 hd590300_1 conda-forge libxml2 2.13.8 h4bc477f_0 conda-forge libzlib 1.3.1 hb9d3cd8_2 conda-forge llvm-openmp 15.0.7 h0cdce71_0 conda-forge lz4-c 1.9.4 h6a678d5_1
markdown-it-py 3.0.0 pypi_0 pypi markupsafe 3.0.2 py310h5eee18b_0
matplotlib 3.10.3 pypi_0 pypi matplotlib-inline 0.1.7 pyhd8ed1ab_1 conda-forge mdurl 0.1.2 pypi_0 pypi mistune 3.1.3 pypi_0 pypi mkl 2021.4.0 h06a4308_640
mkl-service 2.4.0 py310ha2c4b55_0 conda-forge mkl_fft 1.3.1 py310h2b4bcf5_1 conda-forge mkl_random 1.2.2 py310h00e6091_0
mmengine 0.10.7 pypi_0 pypi mpc 1.3.1 h5eee18b_0
mpfr 4.2.1 h5eee18b_0
mpi4py-mpich 3.1.5 pypi_0 pypi mpmath 1.3.0 py310h06a4308_0
msgpack 1.1.0 pypi_0 pypi narwhals 1.39.0 pypi_0 pypi nbclient 0.10.2 pypi_0 pypi nbconvert 7.16.6 pypi_0 pypi nbformat 5.10.4 pypi_0 pypi ncurses 6.5 h2d0b736_3 conda-forge nest-asyncio 1.6.0 pyhd8ed1ab_1 conda-forge nettle 3.9.1 h7ab15ed_0 conda-forge networkx 3.4.2 py310h06a4308_0
ninja 1.11.1.4 pypi_0 pypi nltk 3.9.1 pypi_0 pypi notebook 7.4.2 pypi_0 pypi notebook-shim 0.2.4 pypi_0 pypi numpy 1.24.3 py310hd5efca6_0
numpy-base 1.24.3 py310h8e6c178_0
nvidia-ml-py 12.575.51 pypi_0 pypi ocl-icd 2.3.2 h5eee18b_1
openai 1.78.1 pypi_0 pypi opencv-python 4.11.0.86 pypi_0 pypi openh264 2.3.1 hcb278e6_2 conda-forge openjpeg 2.5.0 hfec8fc6_2 conda-forge openpyxl 3.1.5 pypi_0 pypi openssl 3.5.0 h7b32b05_1 conda-forge overrides 7.7.0 pypi_0 pypi p11-kit 0.24.1 hc5aa10d_0 conda-forge packaging 24.2 pypi_0 pypi pandas 2.2.3 pypi_0 pypi pandocfilters 1.5.1 pypi_0 pypi parso 0.8.4 pyhd8ed1ab_1 conda-forge peft 0.4.0 pypi_0 pypi pexpect 4.9.0 pyhd8ed1ab_1 conda-forge phx-class-registry 4.1.0 pypi_0 pypi pickleshare 0.7.5 pyhd8ed1ab_1004 conda-forge pillow 11.2.1 pypi_0 pypi pip 25.1.1 pyh8b19718_0 conda-forge platformdirs 4.3.8 pyhe01879c_0 conda-forge prometheus-client 0.21.1 pypi_0 pypi prompt-toolkit 3.0.51 pyha770c72_0 conda-forge psutil 7.0.0 py310ha75aee5_0 conda-forge pthread-stubs 0.4 hb9d3cd8_1002 conda-forge ptyprocess 0.7.0 pyhd8ed1ab_1 conda-forge pure_eval 0.2.3 pyhd8ed1ab_1 conda-forge py-cpuinfo 9.0.0 pypi_0 pypi pyarrow 20.0.0 pypi_0 pypi pycparser 2.22 pypi_0 pypi pydantic 2.11.4 pypi_0 pypi pydantic-core 2.33.2 pypi_0 pypi pydeck 0.9.1 pypi_0 pypi pygments 2.19.1 pyhd8ed1ab_0 conda-forge pyparsing 3.2.3 pypi_0 pypi pysocks 1.7.1 py310h06a4308_0
python 3.10.17 hd6af730_0_cpython conda-forge python-dateutil 2.9.0.post0 pyhff2d567_1 conda-forge python-dotenv 1.1.0 pypi_0 pypi python-json-logger 3.3.0 pypi_0 pypi python_abi 3.10 7_cp310 conda-forge pytorch 2.5.1 py3.10_cuda12.4_cudnn9.1.0_0 pytorch pytorch-cuda 12.4 hc786d27_7 pytorch pytorch-mutex 1.0 cuda pytorch pytz 2025.2 pypi_0 pypi pyyaml 6.0.2 py310h5eee18b_0
pyzmq 26.4.0 py310h71f11fc_0 conda-forge readline 8.2 h8c095d6_2 conda-forge referencing 0.36.2 pypi_0 pypi requests 2.32.3 py310h06a4308_1
rfc3339-validator 0.1.4 pypi_0 pypi rfc3986-validator 0.1.1 pypi_0 pypi rich 14.0.0 pypi_0 pypi rpds-py 0.24.0 pypi_0 pypi safetensors 0.5.3 pypi_0 pypi scikit-image 0.25.2 pypi_0 pypi scipy 1.15.3 pypi_0 pypi send2trash 1.8.3 pypi_0 pypi sentencepiece 0.2.0 pypi_0 pypi setuptools 80.1.0 pyhff2d567_0 conda-forge sgmllib3k 1.0.0 pypi_0 pypi shtab 1.7.2 pypi_0 pypi six 1.17.0 pyhd8ed1ab_0 conda-forge sniffio 1.3.1 pypi_0 pypi socksio 1.0.0 pypi_0 pypi soupsieve 2.7 pypi_0 pypi sqlite 3.49.2 h9eae976_0 conda-forge stack_data 0.6.3 pyhd8ed1ab_1 conda-forge streamlit 1.45.1 pypi_0 pypi sty 1.0.6 pypi_0 pypi svt-av1 1.4.1 hcb278e6_0 conda-forge sympy 1.13.1 pypi_0 pypi tbb 2021.8.0 hdb19cb5_0
tenacity 9.1.2 pypi_0 pypi terminado 0.18.1 pypi_0 pypi tifffile 2025.5.10 pypi_0 pypi tiktoken 0.9.0 pypi_0 pypi timeout-decorator 0.5.0 pypi_0 pypi timm 1.0.15 pypi_0 pypi tinycss2 1.4.0 pypi_0 pypi tk 8.6.13 noxft_h4845f30_101 conda-forge tokenizers 0.20.3 pypi_0 pypi toml 0.10.2 pypi_0 pypi tomli 2.2.1 pypi_0 pypi torchaudio 2.5.1 py310_cu124 pytorch torchtriton 3.1.0 py310 pytorch torchvision 0.20.1 py310_cu124 pytorch tornado 6.4.2 py310ha75aee5_0 conda-forge tqdm 4.67.1 pypi_0 pypi traitlets 5.14.3 pyhd8ed1ab_1 conda-forge transformers 4.45.1 pypi_0 pypi transformers-stream-generator 0.0.5 pypi_0 pypi trl 0.10.1 pypi_0 pypi typeguard 4.4.2 pypi_0 pypi types-python-dateutil 2.9.0.20241206 pypi_0 pypi typing-inspection 0.4.0 pypi_0 pypi typing_extensions 4.13.2 pyh29332c3_0 conda-forge tyro 0.9.22 pypi_0 pypi tzdata 2025b h78e105d_0 conda-forge uri-template 1.3.0 pypi_0 pypi urllib3 2.3.0 py310h06a4308_0
validators 0.35.0 pypi_0 pypi watchdog 6.0.0 pypi_0 pypi wcwidth 0.2.13 pyhd8ed1ab_1 conda-forge webcolors 24.11.1 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 1.8.0 pypi_0 pypi wheel 0.45.1 pyhd8ed1ab_1 conda-forge widgetsnbextension 4.0.14 pypi_0 pypi x264 1!164.3095 h166bdaf_2 conda-forge x265 3.5 h924138e_3 conda-forge xlsxwriter 3.2.3 pypi_0 pypi xorg-fixesproto 5.0 hb9d3cd8_1003 conda-forge xorg-kbproto 1.0.7 hb9d3cd8_1003 conda-forge xorg-libx11 1.8.4 h0b41bf4_0 conda-forge xorg-libxau 1.0.12 hb9d3cd8_0 conda-forge xorg-libxdmcp 1.1.5 hb9d3cd8_0 conda-forge xorg-libxext 1.3.4 h0b41bf4_2 conda-forge xorg-libxfixes 5.0.3 h7f98852_1004 conda-forge xorg-xextproto 7.3.0 hb9d3cd8_1004 conda-forge xorg-xproto 7.0.31 hb9d3cd8_1008 conda-forge xtuner 0.1.23 pypi_0 pypi xz 5.8.1 hbcc6ac9_1 conda-forge xz-gpl-tools 5.8.1 hbcc6ac9_1 conda-forge xz-tools 5.8.1 hb9d3cd8_1 conda-forge yaml 0.2.5 h7b6447c_0
yapf 0.43.0 pypi_0 pypi zeromq 4.3.5 h3b0a872_7 conda-forge zipp 3.21.0 pyhd8ed1ab_1 conda-forge zlib 1.3.1 hb9d3cd8_2 conda-forge zstd 1.5.7 hb8e6e7a_2 conda-forge

Reproduction

export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 export USE_COT="1" torchrun --master_port=9989 --nproc-per-node=8 run.py --data AI2D_TEST --model InternVL3-38B

Environment

sys.platform: linux
Python: 3.10.17 | packaged by conda-forge | (main, Apr 10 2025, 22:19:12) [GCC 13.3.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1,2,3,4,5,6,7: NVIDIA A100 80GB PCIe
CUDA_HOME: /usr/local/cuda-12.4
NVCC: Cuda compilation tools, release 12.4, V12.4.131
GCC: gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0
PyTorch: 2.5.1
PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 12.4
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 90.1
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 

TorchVision: 0.20.1
LMDeploy: 0.8.0+50446de
transformers: 4.45.1
gradio: Not Found
fastapi: 0.115.12
pydantic: 2.11.4
triton: 3.1.0
NVIDIA Topology: 
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV12    PXB     PXB     SYS     SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU1    NV12     X      PXB     PXB     SYS     SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU2    PXB     PXB      X      NV12    SYS     SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU3    PXB     PXB     NV12     X      SYS     SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU4    SYS     SYS     SYS     SYS      X      NV12    PXB     PXB     PXB     32-63,96-127    1               N/A
GPU5    SYS     SYS     SYS     SYS     NV12     X      PXB     PXB     PIX     32-63,96-127    1               N/A
GPU6    SYS     SYS     SYS     SYS     PXB     PXB      X      NV12    PXB     32-63,96-127    1               N/A
GPU7    SYS     SYS     SYS     SYS     PXB     PXB     NV12     X      PXB     32-63,96-127    1               N/A
NIC0    SYS     SYS     SYS     SYS     PXB     PIX     PXB     PXB      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0

Error traceback

avdravid avatar May 30 '25 16:05 avdravid

We only set USE_COT=1 during the evaluation of reasoning benchmarks. Tasks focusing on perception abilities place limited demands on reasoning, so enabling CoT may not always lead to performance improvements. Additionally, we observe that evaluating MMVet with CoT leads to inferior performance, which seems to be an issue with its judging method.

Weiyun1025 avatar Aug 30 '25 03:08 Weiyun1025

Have you solved this problem? I also meet it.

Graysonicc avatar Sep 15 '25 14:09 Graysonicc