opencompass icon indicating copy to clipboard operation
opencompass copied to clipboard

[FIX] Added handling for the "begin section" in meta_template to APITemplateParser

Open hailsham opened this issue 6 months ago • 0 comments

Motivation

This is a fix for the issue. This issue describes a problem where specifying a system_prompt in the meta_template does not work as expected because APITemplateParser does not handle the content within the begin section of the meta_template. This seems to fall short of user expectations. When users write the meta_template in the following manner, they likely expect the system_prompt to be added to the prompt list.

_meta_template=dict(
    begin=dict(role='SYSTEM', prompt='You are a cat'),
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True)
    ],
)

Expected behavior:

"origin_prompt": [
  {
      "prompt": "You are a cat.",
      "role": "SYSTEM"
  },
  {
      "role": "HUMAN",
      "prompt": "Question: Angelo and Melanie want to plan how many hours over the next week they should study together for their test next week. They have 2 chapters of their textbook to study and 4 worksheets to memorize. They figure out that they should dedicate 3 hours to each chapter of their textbook and 1.5 hours for each worksheet. If they plan to study no more than 4 hours each day, how many days should they plan to study total over the next week if they take a 10-minute break every hour, include 3 10-minute snack breaks each day, and 30 minutes for lunch each day?\nLet's think step by step\nAnswer:"
  },
  ... 
]

Actual behavior:

"origin_prompt": [
  {
      "role": "HUMAN",
      "prompt": "Question: Angelo and Melanie want to plan how many hours over the next week they should study together for their test next week. They have 2 chapters of their textbook to study and 4 worksheets to memorize. They figure out that they should dedicate 3 hours to each chapter of their textbook and 1.5 hours for each worksheet. If they plan to study no more than 4 hours each day, how many days should they plan to study total over the next week if they take a 10-minute break every hour, include 3 10-minute snack breaks each day, and 30 minutes for lunch each day?\nLet's think step by step\nAnswer:"
  },
  ... 
]

Modification

The handling of the begin section in meta_template has been added to APITemplateParser, similar to the handling in LMTemplateParser.

Checklist

Before PR:

  • [x] Pre-commit or other linting tools are used to fix the potential lint issues.
  • [x] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
  • [x] The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  • [ ] The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

  • [ ] If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
  • [ ] CLA has been signed and all committers have signed the CLA in this PR.

hailsham avatar Aug 08 '24 05:08 hailsham