[Bug] 评测ChemBench时导入数据集报错

[lzpp2598]

先决条件

• [x] 我已经搜索过 问题 和 讨论 但未得到预期的帮助。
• [x] 错误在 最新版本 中尚未被修复。

问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

环境

{'CUDA available': False, 'GCC': 'gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0', 'MMEngine': '0.10.7', 'MUSA available': False, 'OpenCV': '4.11.0', 'PyTorch': '2.7.1+cu126', 'PyTorch compiling details': 'PyTorch built with:\n' ' - GCC 11.2\n' ' - C++ Version: 201703\n' ' - Intel(R) oneAPI Math Kernel Library Version ' '2024.2-Product Build 20240605 for Intel(R) 64 ' 'architecture applications\n' ' - Intel(R) MKL-DNN v3.7.1 (Git Hash ' '8d263e693366ef8db40acc569cc7d8edf644556d)\n' ' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n' ' - LAPACK is enabled (usually provided by ' 'MKL)\n' ' - NNPACK is enabled\n' ' - CPU capability usage: AVX512\n' ' - Build settings: BLAS_INFO=mkl, ' 'BUILD_TYPE=Release, ' 'COMMIT_SHA=e2d141dbde55c2a4370fac5165b0561b6af4798b, ' 'CUDA_VERSION=12.6, CUDNN_VERSION=9.5.1, ' 'CXX_COMPILER=/opt/rh/gcc-toolset-11/root/usr/bin/c++, ' 'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 ' '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL ' '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER ' '-DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM ' '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK ' '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC ' '-Wall -Wextra -Werror=return-type ' '-Werror=non-virtual-dtor ' '-Werror=range-loop-construct ' '-Werror=bool-operation -Wnarrowing ' '-Wno-missing-field-initializers ' '-Wno-unknown-pragmas -Wno-unused-parameter ' '-Wno-strict-overflow -Wno-strict-aliasing ' '-Wno-stringop-overflow -Wsuggest-override ' '-Wno-psabi -Wno-error=old-style-cast ' '-fdiagnostics-color=always -faligned-new ' '-Wno-maybe-uninitialized -fno-math-errno ' '-fno-trapping-math -Werror=format ' '-Wno-stringop-overflow, LAPACK_INFO=mkl, ' 'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, ' 'TORCH_VERSION=2.7.1, USE_CUDA=ON, USE_CUDNN=ON, ' 'USE_CUSPARSELT=1, USE_GFLAGS=OFF, USE_GLOG=OFF, ' 'USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, ' 'USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, ' 'USE_OPENMP=ON, USE_ROCM=OFF, ' 'USE_ROCM_KERNEL_ASSERT=OFF, \n', 'Python': '3.10.18 (main, Jun 5 2025, 13:14:17) [GCC 11.2.0]', 'lmdeploy': "not installed:No module named 'lmdeploy'", 'numpy_random_seed': 2147483648, 'opencompass': '0.4.2+c29c258', 'sys.platform': 'linux', 'transformers': '4.52.4'}

重现问题 - 代码/配置示例

from mmengine.config import read_base from opencompass.partitioners import SizePartitioner, NaivePartitioner from opencompass.runners import LocalRunner from opencompass.tasks import OpenICLInferTask, OpenICLEvalTask

with read_base(): from opencompass.configs.datasets.ChemBench.ChemBench_gen_a9f753 import
chembench_datasets from opencompass.models import OpenAISDK

model_name = "Qwen3-8B" model_field = "Qwen3-8B" model_api_url = "http://g0003:6065/v1"

datasets = [*chembench_datasets]

models = [ dict( type=OpenAISDK, # 使用 OpenAI 模型 # 以下为 OpenAI 初始化参数 path='Qwen3-8B', # 指定模型类型 openai_api_base=model_api_url, # 服务地址 key='EMPTY', # OpenAI API Key max_seq_len=32768, # 最大输入长度 temperature=0.6, # 控制生成文本的随机性 extra_body=dict( top_p=0.95, # 控制生成文本的多样性 ), # 以下参数为各类模型都有的参数，非 OpenAI 的初始化参数 abbr='Qwen3-8B', # 模型简称 # run_cfg=dict(num_gpus=0), # 资源需求（不需要 GPU） max_out_len=16000, # 最长生成长度 batch_size=1, # 批次大小

),

]

infer = dict( partitioner=dict(type=SizePartitioner, max_task_size=500), runner=dict(type=LocalRunner, max_num_workers=100, task=dict(type=OpenICLInferTask)), )

eval with local runner

eval = dict( partitioner=dict(type=NaivePartitioner, n=100), runner=dict( type=LocalRunner, max_num_workers=16, task=dict(type=OpenICLEvalTask) ), )

work_dir = f"outputs/{model_name}/"

重现问题 - 命令或脚本

python run.py examples/eval_chembench.py

重现问题 - 错误信息

Traceback (most recent call last): File "/data/home/zdhs0075/opencompass/run.py", line 4, in main() File "/data/home/zdhs0075/opencompass/opencompass/cli/main.py", line 259, in main cfg = get_config_from_arg(args) File "/data/home/zdhs0075/opencompass/opencompass/utils/run.py", line 97, in get_config_from_arg config = Config.fromfile(args.config, format_python_code=False) File "/data/home/zdhs0075/miniforge3/envs/opencompass_2/lib/python3.10/site-packages/mmengine/config/config.py", line 494, in fromfile raise e File "/data/home/zdhs0075/miniforge3/envs/opencompass_2/lib/python3.10/site-packages/mmengine/config/config.py", line 492, in fromfile cfg_dict, imported_names = Config._parse_lazy_import(filename) File "/data/home/zdhs0075/miniforge3/envs/opencompass_2/lib/python3.10/site-packages/mmengine/config/config.py", line 1074, in _parse_lazy_import raise ConfigParsingError( mmengine.config.utils.ConfigParsingError: /data/home/zdhs0075/miniforge3/envs/opencompass_2/lib/python3.10/site-packages/opencompass/configs/datasets/ChemBench/ChemBench_gen_a9f753.py not found! It means that incorrect module is defined in with read_base(): = from opencompass.configs.datasets.ChemBench.ChemBench_gen_a9f753 import ..., please make sure the base config module is valid and is consistent with the prior import logic

其他信息

No response

[tonysy]

Please make sure you have installed the opencompass successfully in your environment.

[lzpp2598]

I confirm that OpenCompass has been successfully installed. I tested and found that when importing other datasets (such as AIME2024), this error does not occur.

[kamieee]

I guess you forgot to download the dataset/didn't use the dataset correctly. Because ChemBench doesn't belong in the same path as AIME2024. wget https://github.com/open-compass/opencompass/releases/download/0.2.2.rc1/OpenCompassData-core-20240207.zip