opencompass icon indicating copy to clipboard operation
opencompass copied to clipboard

请问我想对cmmlu这类选择题的模板进行更改,但是system变量无法赋值

Open Sweetclover opened this issue 2 years ago • 1 comments

Prerequisite

Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

cmmlu_infer_cfg = dict(

     ice_template=dict(
          type=PromptTemplate,
          template=dict(
              begin=[dict(role='SYSTEM', fallback_role='HUMAN', prompt='以下是关于{_ch_name}的单项选择题,请直接给出正确答案的选项。'),'</E>'],
             #begin="</E>",
              round=[
                  dict(
                      role="HUMAN",
                      prompt=
                      f"USER: {{question}}\nA. {{A}}\nB. {{B}}\nC. {{C}}\nD. {{D}}"
                  ),
                  dict(role="BOT", prompt=' ASSISTANT: 答案是: {answer}</s>'),
              ]),
          ice_token="</E>",
     ),

)

Reproduces the problem - code/configuration sample

请问怎么让begin里的_ch_name进行赋值呢,我看只有round里面会赋值。或者我应该怎么写比较好呢。”以下是关于{_ch_name}的单项选择题,请直接给出正确答案的选项“在fewshot中只在最前面出现一次

Reproduces the problem - command or script

1

Reproduces the problem - error message

1

Other information

1

Sweetclover avatar Oct 08 '23 03:10 Sweetclover

我希望的模板是 以下是关于{法律}的单项选择题,请直接给出正确答案的选项。 {qustion1} {A} {B} {C} {D} 答案是:{answer} {qustion2} {A} {B} {C} {D} 答案是:{answer} ''' {question} {A} {B} {C} {D} 答案是:

Sweetclover avatar Oct 08 '23 08:10 Sweetclover