eval-scope icon indicating copy to clipboard operation
eval-scope copied to clipboard

使用opencompass backend结果没有分数

Open Molri19 opened this issue 7 months ago • 3 comments

问题描述 / Issue Description

请简要描述您遇到的问题。 / Please briefly describe the issue you encountered.

EvalScope 版本 / Version (必须)

v0.xx.x

使用的工具 / Tools Used

  • [ ] Native / 原生框架
  • [* ] Opencompass backend
  • [ ] VLMEvalKit backend
  • [] RAGEval backend
  • [ ] Perf / 模型推理压测工具
  • [ ] Arena /竞技场模式

执行的代码或指令 / Code or Commands Executed

task_cfg_dict = dict( eval_backend='OpenCompass', eval_type='service', eval_batch_size='100', stage='review', eval_config={ 'datasets': ['mmlu'], 'models': [ {'path': '/data00/models/deepseek-r1-w4a8', 'openai_api_base': 'http://221.194.152.47:8000/v1/chat/completions', 'is_chat': True, 'batch_size': 100 }, ], 'work_dir': 'outputs/deepseek-r1-w4a8', 'limit': None, }, ) #%% from evalscope.run import run_task from evalscope.summarizer import Summarizer def run_eval(): # 选项 1: python 字典 task_cfg = task_cfg_dict # 选项 2: yaml 配置文件 # task_cfg = 'eval_openai_api.yaml' # 选项 3: json 配置文件 # task_cfg = 'eval_openai_api.json' run_task(task_cfg=task_cfg) print('>> Start to get the report with summarizer ...') report_list = Summarizer.get_report_from_cfg(task_cfg) print(f'\n>> The report list: {report_list}') run_eval()

请提供您执行的主要代码或指令。 / Please provide the main code or commands you executed. 例如 / For example:

# 例如:执行的Python代码 / Python code executed
import some_module

some_module.some_function()
# 例如:在终端中执行的指令 / Terminal command executed
python script.py

错误日志 / Error Log

请粘贴完整的错误日志或控制台输出。 / Please paste the full error log or console output. 例如 / For example:

dataset version metric mode /data00/models/deepseek-r1-w4a8
lukaemon_mmlu_college_biology - - - -
lukaemon_mmlu_college_chemistry - - - -
lukaemon_mmlu_college_computer_science - - - -
lukaemon_mmlu_college_mathematics - - - -
lukaemon_mmlu_college_physics - - - -
lukaemon_mmlu_electrical_engineering - - - -
lukaemon_mmlu_astronomy - - - -
lukaemon_mmlu_anatomy - - - -
lukaemon_mmlu_abstract_algebra - - - -
lukaemon_mmlu_machine_learning - - - -
lukaemon_mmlu_clinical_knowledge - - - -
lukaemon_mmlu_global_facts - - - -
lukaemon_mmlu_management - - - -
lukaemon_mmlu_nutrition - - - -
lukaemon_mmlu_marketing - - - -
lukaemon_mmlu_professional_accounting - - - -
lukaemon_mmlu_high_school_geography - - - -
lukaemon_mmlu_international_law - - - -
lukaemon_mmlu_moral_scenarios - - - -
lukaemon_mmlu_computer_security - - - -
lukaemon_mmlu_high_school_microeconomics - - - -
lukaemon_mmlu_professional_law - - - -
lukaemon_mmlu_medical_genetics - - - -
lukaemon_mmlu_professional_psychology - - - -
lukaemon_mmlu_jurisprudence - - - -
lukaemon_mmlu_world_religions - - - -
lukaemon_mmlu_philosophy - - - -
lukaemon_mmlu_virology - - - -
lukaemon_mmlu_high_school_chemistry - - - -
lukaemon_mmlu_public_relations - - - -
lukaemon_mmlu_high_school_macroeconomics - - - -
lukaemon_mmlu_human_sexuality - - - -
lukaemon_mmlu_elementary_mathematics - - - -
lukaemon_mmlu_high_school_physics - - - -
lukaemon_mmlu_high_school_computer_science - - - -
lukaemon_mmlu_high_school_european_history - - - -
lukaemon_mmlu_business_ethics - - - -
lukaemon_mmlu_moral_disputes - - - -
lukaemon_mmlu_high_school_statistics - - - -
lukaemon_mmlu_miscellaneous - - - -
lukaemon_mmlu_formal_logic - - - -
lukaemon_mmlu_high_school_government_and_politics - - - -
lukaemon_mmlu_prehistory - - - -
lukaemon_mmlu_security_studies - - - -
lukaemon_mmlu_high_school_biology - - - -
lukaemon_mmlu_logical_fallacies - - - -
lukaemon_mmlu_high_school_world_history - - - -
lukaemon_mmlu_professional_medicine - - - -
lukaemon_mmlu_high_school_mathematics - - - -
lukaemon_mmlu_college_medicine - - - -
lukaemon_mmlu_high_school_us_history - - - -
lukaemon_mmlu_sociology - - - -
lukaemon_mmlu_econometrics - - - -
lukaemon_mmlu_high_school_psychology - - - -
lukaemon_mmlu_human_aging - - - -
lukaemon_mmlu_us_foreign_policy - - - -
lukaemon_mmlu_conceptual_physics - - - -
Traceback (most recent call last):
  File "script.py", line 10, in <module>
    main()
  File "script.py", line 5, in main
    do_something()
  File "some_module.py", line 20, in do_something
    raise ValueError("An error occurred")
ValueError: An error occurred

运行环境 / Runtime Environment

  • 操作系统 / Operating System:

    • [ ] Windows
    • [ ] macOS
    • [*] Ubuntu
  • Python版本 / Python Version:

    • [ ] 3.11
    • [ ] 3.10
    • [ ] 3.9

其他信息 / Additional Information

是不是和path填的本地路径有关系,我在该路径下发现有这些文件:

Image

如果有其他相关信息,请在此处提供。 / If there is any other relevant information, please provide it here.

Molri19 avatar Apr 30 '25 02:04 Molri19

使用的ms-opencompass 版本是多少呢,目前是0.1.6 版本,看输出结果对不上呀

Yunnglin avatar May 07 '25 12:05 Yunnglin

问题解决了吗?我的也是打印得空的

winni0 avatar May 19 '25 07:05 winni0

是path填的本地路径的原因,这里建议只填模型名称

Yunnglin avatar May 19 '25 10:05 Yunnglin

感谢你的反馈!我们将关闭此问题。如果您有任何疑问,请随时重新打开它。如果EvalScope对您有所帮助,欢迎给我们点个STAR以示支持,谢谢!

Yunnglin avatar Jun 16 '25 06:06 Yunnglin