open-compass
[Bug] xunfei api 评测报错
先决条件
问题类型
我正在使用官方支持的任务/模型/数据集进行评估。
环境
Ubuntu 22.04 miniconda3 opencompass[api]
重现问题 - 代码/配置示例
import os from opencompass.models import XunFeiSpark, XunFei from opencompass.partitioners import NaivePartitioner from opencompass.runners.local_api import LocalAPIRunner from opencompass.tasks import OpenICLInferTask
你前面获得的 api 服务地址
model_url = 'wss://spark-api.xf-yun.com/v1.1/chat' # lite api_key = os.getenv('X_API_KEY') api_meta_template = dict( round=[ dict(role='HUMAN', api_role='HUMAN'), dict(role='BOT', api_role='BOT', generate=True), ], reserved_roles=[dict(role='SYSTEM', api_role='SYSTEM'),] )
models = [ dict( type=XunFeiSpark, path='lite', # 请求服务时的 model name url=model_url, # 服务地址 app_id='xxx', api_key='xxx', api_secret=api_key, # API key query_per_second=0.3, # 服务请求速率 max_seq_len=4096, # 最大输入长度 # max_out_len=1024, # 最大输出长度 # meta_template=api_meta_template, # rpm_verbose=True, # 是否打印请求速率 # temperature=0.1, # 生成温度 # batch_size=1, # 批处理大小 retry=2, # 重试次数 # extra_body={"stream": False}, # logprobs = True, # verbose=True )]
infer = dict( partitioner=dict(type=NaivePartitioner), runner=dict( type=LocalAPIRunner, max_num_workers=2, concurrent_users=1, task=dict(type=OpenICLInferTask)), )
重现问题 - 命令或脚本
opencompass --models xunfei_api --datasets demo_gsm8k_chat_gen --debug
重现问题 - 错误信息
(opencompass) wtl@iZf8z1mesjci0f2gxm0hkbZ:~/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/configs$ opencompass --models xunfei_api --datasets demo_math_base_gen --debug
03/05 09:52:40 - OpenCompass - INFO - Loading demo_math_base_gen: /home/wtl/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/configs/./datasets/demo/demo_math_base_gen.py
03/05 09:52:40 - OpenCompass - INFO - Loading xunfei_api: /home/wtl/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/configs/./models/xunfei/xunfei_api.py
03/05 09:52:40 - OpenCompass - INFO - Loading example: /home/wtl/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/configs/./summarizers/example.py
03/05 09:52:40 - OpenCompass - INFO - Current exp folder: outputs/default/20250305_095240
03/05 09:52:40 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
03/05 09:52:40 - OpenCompass - INFO - Partitioned into 1 tasks.
03/05 09:52:41 - OpenCompass - INFO - Task [X-lite/demo_math]
03/05 09:52:41 - OpenCompass - INFO - Try to load the data from /home/wtl/.cache/opencompass/./data/math/
03/05 09:52:41 - OpenCompass - INFO - Start inferencing [X-lite/demo_math]
[2025-03-05 09:52:42,017] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2025-03-05 09:52:42,017] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
0%| | 0/64 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/wtl/miniconda3/envs/opencompass/bin/opencompass", line 8, in
其他信息
使用xunfei api 模式评测报错
We will have a look into the problem. For the moment, please try to use OpenAISDK for the Xunfei model evaluation, it will be a more general solution for API model evaluation.
