opencompass
opencompass copied to clipboard
[Feature] AGIEval数据集是否可以支持few-shot评测设置?
描述该功能
opencompass/opencompass/datasets/agieval/agieval.py 文件显示,目前AGIEval评测仅支持zero-shot设置:
def load(path: str, name: str, setting_name: str):
from .dataset_loader import load_dataset, load_dataset_as_result_schema
assert setting_name in 'zero-shot', 'only support zero-shot setting'
dataset_wo_label = load_dataset(name, setting_name, path)
dataset_with_label = load_dataset_as_result_schema(name, path)
dataset = []
for d1, d2 in zip(dataset_wo_label, dataset_with_label):
dataset.append({
'id': d2.index,
'problem_input': d1['context'],
'label': d2.label,
})
dataset = Dataset.from_list(dataset)
return dataset
原生AGIEval支持few-shot设置,是否可以追加few-shot评测设置?
是否希望自己实现该功能?
- [ ] 我希望自己来实现这一功能,并向 OpenCompass 贡献代码!