使用opencompass backend结果没有分数
问题描述 / Issue Description
请简要描述您遇到的问题。 / Please briefly describe the issue you encountered.
EvalScope 版本 / Version (必须)
v0.xx.x
使用的工具 / Tools Used
- [ ] Native / 原生框架
- [* ] Opencompass backend
- [ ] VLMEvalKit backend
- [] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena /竞技场模式
执行的代码或指令 / Code or Commands Executed
task_cfg_dict = dict( eval_backend='OpenCompass', eval_type='service', eval_batch_size='100', stage='review', eval_config={ 'datasets': ['mmlu'], 'models': [ {'path': '/data00/models/deepseek-r1-w4a8', 'openai_api_base': 'http://221.194.152.47:8000/v1/chat/completions', 'is_chat': True, 'batch_size': 100 }, ], 'work_dir': 'outputs/deepseek-r1-w4a8', 'limit': None, }, ) #%% from evalscope.run import run_task from evalscope.summarizer import Summarizer def run_eval(): # 选项 1: python 字典 task_cfg = task_cfg_dict # 选项 2: yaml 配置文件 # task_cfg = 'eval_openai_api.yaml' # 选项 3: json 配置文件 # task_cfg = 'eval_openai_api.json' run_task(task_cfg=task_cfg) print('>> Start to get the report with summarizer ...') report_list = Summarizer.get_report_from_cfg(task_cfg) print(f'\n>> The report list: {report_list}') run_eval()
请提供您执行的主要代码或指令。 / Please provide the main code or commands you executed. 例如 / For example:
# 例如:执行的Python代码 / Python code executed
import some_module
some_module.some_function()
# 例如:在终端中执行的指令 / Terminal command executed
python script.py
错误日志 / Error Log
请粘贴完整的错误日志或控制台输出。 / Please paste the full error log or console output. 例如 / For example:
| dataset | version | metric | mode | /data00/models/deepseek-r1-w4a8 |
|---|---|---|---|---|
| lukaemon_mmlu_college_biology | - | - | - | - |
| lukaemon_mmlu_college_chemistry | - | - | - | - |
| lukaemon_mmlu_college_computer_science | - | - | - | - |
| lukaemon_mmlu_college_mathematics | - | - | - | - |
| lukaemon_mmlu_college_physics | - | - | - | - |
| lukaemon_mmlu_electrical_engineering | - | - | - | - |
| lukaemon_mmlu_astronomy | - | - | - | - |
| lukaemon_mmlu_anatomy | - | - | - | - |
| lukaemon_mmlu_abstract_algebra | - | - | - | - |
| lukaemon_mmlu_machine_learning | - | - | - | - |
| lukaemon_mmlu_clinical_knowledge | - | - | - | - |
| lukaemon_mmlu_global_facts | - | - | - | - |
| lukaemon_mmlu_management | - | - | - | - |
| lukaemon_mmlu_nutrition | - | - | - | - |
| lukaemon_mmlu_marketing | - | - | - | - |
| lukaemon_mmlu_professional_accounting | - | - | - | - |
| lukaemon_mmlu_high_school_geography | - | - | - | - |
| lukaemon_mmlu_international_law | - | - | - | - |
| lukaemon_mmlu_moral_scenarios | - | - | - | - |
| lukaemon_mmlu_computer_security | - | - | - | - |
| lukaemon_mmlu_high_school_microeconomics | - | - | - | - |
| lukaemon_mmlu_professional_law | - | - | - | - |
| lukaemon_mmlu_medical_genetics | - | - | - | - |
| lukaemon_mmlu_professional_psychology | - | - | - | - |
| lukaemon_mmlu_jurisprudence | - | - | - | - |
| lukaemon_mmlu_world_religions | - | - | - | - |
| lukaemon_mmlu_philosophy | - | - | - | - |
| lukaemon_mmlu_virology | - | - | - | - |
| lukaemon_mmlu_high_school_chemistry | - | - | - | - |
| lukaemon_mmlu_public_relations | - | - | - | - |
| lukaemon_mmlu_high_school_macroeconomics | - | - | - | - |
| lukaemon_mmlu_human_sexuality | - | - | - | - |
| lukaemon_mmlu_elementary_mathematics | - | - | - | - |
| lukaemon_mmlu_high_school_physics | - | - | - | - |
| lukaemon_mmlu_high_school_computer_science | - | - | - | - |
| lukaemon_mmlu_high_school_european_history | - | - | - | - |
| lukaemon_mmlu_business_ethics | - | - | - | - |
| lukaemon_mmlu_moral_disputes | - | - | - | - |
| lukaemon_mmlu_high_school_statistics | - | - | - | - |
| lukaemon_mmlu_miscellaneous | - | - | - | - |
| lukaemon_mmlu_formal_logic | - | - | - | - |
| lukaemon_mmlu_high_school_government_and_politics | - | - | - | - |
| lukaemon_mmlu_prehistory | - | - | - | - |
| lukaemon_mmlu_security_studies | - | - | - | - |
| lukaemon_mmlu_high_school_biology | - | - | - | - |
| lukaemon_mmlu_logical_fallacies | - | - | - | - |
| lukaemon_mmlu_high_school_world_history | - | - | - | - |
| lukaemon_mmlu_professional_medicine | - | - | - | - |
| lukaemon_mmlu_high_school_mathematics | - | - | - | - |
| lukaemon_mmlu_college_medicine | - | - | - | - |
| lukaemon_mmlu_high_school_us_history | - | - | - | - |
| lukaemon_mmlu_sociology | - | - | - | - |
| lukaemon_mmlu_econometrics | - | - | - | - |
| lukaemon_mmlu_high_school_psychology | - | - | - | - |
| lukaemon_mmlu_human_aging | - | - | - | - |
| lukaemon_mmlu_us_foreign_policy | - | - | - | - |
| lukaemon_mmlu_conceptual_physics | - | - | - | - |
Traceback (most recent call last):
File "script.py", line 10, in <module>
main()
File "script.py", line 5, in main
do_something()
File "some_module.py", line 20, in do_something
raise ValueError("An error occurred")
ValueError: An error occurred
运行环境 / Runtime Environment
-
操作系统 / Operating System:
- [ ] Windows
- [ ] macOS
- [*] Ubuntu
-
Python版本 / Python Version:
- [ ] 3.11
- [ ] 3.10
- [ ] 3.9
其他信息 / Additional Information
是不是和path填的本地路径有关系,我在该路径下发现有这些文件:
如果有其他相关信息,请在此处提供。 / If there is any other relevant information, please provide it here.
使用的ms-opencompass 版本是多少呢,目前是0.1.6 版本,看输出结果对不上呀
问题解决了吗?我的也是打印得空的
是path填的本地路径的原因,这里建议只填模型名称
感谢你的反馈!我们将关闭此问题。如果您有任何疑问,请随时重新打开它。如果EvalScope对您有所帮助,欢迎给我们点个STAR以示支持,谢谢!