opencompass icon indicating copy to clipboard operation
opencompass copied to clipboard

[Feature] AGIEval数据集是否可以支持few-shot评测设置?

Open cabbagecabbage opened this issue 9 months ago • 0 comments

描述该功能

opencompass/opencompass/datasets/agieval/agieval.py 文件显示,目前AGIEval评测仅支持zero-shot设置:

def load(path: str, name: str, setting_name: str):
      from .dataset_loader import load_dataset, load_dataset_as_result_schema

      assert setting_name in 'zero-shot', 'only support zero-shot setting'
      dataset_wo_label = load_dataset(name, setting_name, path)
      dataset_with_label = load_dataset_as_result_schema(name, path)
      dataset = []
      for d1, d2 in zip(dataset_wo_label, dataset_with_label):
          dataset.append({
              'id': d2.index,
              'problem_input': d1['context'],
              'label': d2.label,
          })
      dataset = Dataset.from_list(dataset)
      return dataset

原生AGIEval支持few-shot设置,是否可以追加few-shot评测设置?

是否希望自己实现该功能?

  • [ ] 我希望自己来实现这一功能,并向 OpenCompass 贡献代码!

cabbagecabbage avatar May 14 '24 10:05 cabbagecabbage