opencompass icon indicating copy to clipboard operation
opencompass copied to clipboard

[Feature] 自定义数据集命令如何修改评估指标?

Open liushiton opened this issue 1 year ago • 0 comments

Describe the feature

python run.py
--models hf_llama2_7b
--custom-dataset-path xxx/test_qa.jsonl
--custom-dataset-data-type qa
--custom-dataset-infer-method gen

使用这个命令得到的结果得分默认是accuracy。这意味着要完全相同才能算对么?如何替换成别的评估指标呢? 通过新增配置文件,学习成本有点高。。。

Will you implement it?

  • [ ] I would like to implement this feature and create a PR!

liushiton avatar Apr 12 '24 07:04 liushiton