自查清单
在提交 issue 之前，请确保您已完成以下步骤:
- [X] 我已仔细阅读了相关使用说明文档
- [X] 我已查看了常见问题解答
- [ ] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题
问题描述
请简要描述您遇到的问题。
EvalScope 版本（必填）
v1.1.0
使用的工具
- [ ] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [X] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena / 竞技场模式
执行的代码或指令
from evalscope.run import run_task
task_cfg = {
"work_dir": "outputs",
"eval_backend": "RAGEval",
"eval_config": {
"tool": "MTEB",
"model": [
{
"model_name": "youtu-embedding",
"api_base": "http://192.168.0.147:10600/v1",
"api_key": "123",
"dimensions": 2048,
"encode_kwargs": {
"batch_size": 128,
},
}
],
"eval": {
"tasks": ["CustomRetrieval"],
"dataset_path": "/study_project/evalscope/embedding_test/retrieval_data",
"verbosity": 2,
"overwrite_results": True,
"limits": 500,
},
},
}
run_task(task_cfg=task_cfg)
错误日志
Traceback (most recent call last):
File "/DATA/LLM/gaojiale/study_project/evalscope/embedding_test/embedding_test.py", line 28, in
run_task(task_cfg=task_cfg)
File "/DATA/LLM/gaojiale/miniconda/envs/eval/lib/python3.10/site-packages/evalscope/run.py", line 28, in run_task
return run_single_task(task_cfg, run_time)
File "/DATA/LLM/gaojiale/miniconda/envs/eval/lib/python3.10/site-packages/evalscope/run.py", line 39, in run_single_task
result = run_non_native_backend(task_cfg, outputs)
File "/DATA/LLM/gaojiale/miniconda/envs/eval/lib/python3.10/site-packages/evalscope/run.py", line 89, in run_non_native_backend
backend_manager.run()
File "/DATA/LLM/gaojiale/miniconda/envs/eval/lib/python3.10/site-packages/evalscope/backend/rag_eval/backend_manager.py", line 83, in run
self.run_mteb(model_args, eval_args)
File "/DATA/LLM/gaojiale/miniconda/envs/eval/lib/python3.10/site-packages/evalscope/backend/rag_eval/backend_manager.py", line 47, in run_mteb
one_stage_eval(model_args_list[0], eval_args)
File "/DATA/LLM/gaojiale/miniconda/envs/eval/lib/python3.10/site-packages/evalscope/backend/rag_eval/cmteb/task_template.py", line 55, in one_stage_eval
show_results(eval_args['output_folder'], model, results)
File "/DATA/LLM/gaojiale/miniconda/envs/eval/lib/python3.10/site-packages/evalscope/backend/rag_eval/cmteb/task_template.py", line 23, in show_results
'Task Type': main_res.task_type,
File "/DATA/LLM/gaojiale/miniconda/envs/eval/lib/python3.10/site-packages/mteb/load_results/task_results.py", line 229, in task_type
return self.task.metadata.type
File "/DATA/LLM/gaojiale/miniconda/envs/eval/lib/python3.10/functools.py", line 981, in get
val = self.func(instance)
File "/DATA/LLM/gaojiale/miniconda/envs/eval/lib/python3.10/site-packages/mteb/load_results/task_results.py", line 218, in task
return get_task(self.task_name)
File "/DATA/LLM/gaojiale/miniconda/envs/eval/lib/python3.10/site-packages/mteb/overview.py", line 422, in get_task
raise KeyError(suggestion)
KeyError: "KeyError: 'CustomRetrieval' not found. Did you mean: EcomRetrieval?"
运行环境
其他信息
如果有其他相关信息，请在此处提供。
modelscope
