opencompass
opencompass copied to clipboard
[FIX] Added handling for the "begin section" in meta_template to APITemplateParser
Motivation
This is a fix for the issue. This issue describes a problem where specifying a system_prompt
in the meta_template
does not work as expected because APITemplateParser
does not handle the content within the begin section of the meta_template. This seems to fall short of user expectations. When users write the meta_template in the following manner, they likely expect the system_prompt to be added to the prompt list.
_meta_template=dict(
begin=dict(role='SYSTEM', prompt='You are a cat'),
round=[
dict(role='HUMAN', api_role='HUMAN'),
dict(role='BOT', api_role='BOT', generate=True)
],
)
Expected behavior:
"origin_prompt": [
{
"prompt": "You are a cat.",
"role": "SYSTEM"
},
{
"role": "HUMAN",
"prompt": "Question: Angelo and Melanie want to plan how many hours over the next week they should study together for their test next week. They have 2 chapters of their textbook to study and 4 worksheets to memorize. They figure out that they should dedicate 3 hours to each chapter of their textbook and 1.5 hours for each worksheet. If they plan to study no more than 4 hours each day, how many days should they plan to study total over the next week if they take a 10-minute break every hour, include 3 10-minute snack breaks each day, and 30 minutes for lunch each day?\nLet's think step by step\nAnswer:"
},
...
]
Actual behavior:
"origin_prompt": [
{
"role": "HUMAN",
"prompt": "Question: Angelo and Melanie want to plan how many hours over the next week they should study together for their test next week. They have 2 chapters of their textbook to study and 4 worksheets to memorize. They figure out that they should dedicate 3 hours to each chapter of their textbook and 1.5 hours for each worksheet. If they plan to study no more than 4 hours each day, how many days should they plan to study total over the next week if they take a 10-minute break every hour, include 3 10-minute snack breaks each day, and 30 minutes for lunch each day?\nLet's think step by step\nAnswer:"
},
...
]
Modification
The handling of the begin section
in meta_template
has been added to APITemplateParser
, similar to the handling in LMTemplateParser
.
Checklist
Before PR:
- [x] Pre-commit or other linting tools are used to fix the potential lint issues.
- [x] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
- [x] The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
- [ ] The documentation has been modified accordingly, like docstring or example tutorials.
After PR:
- [ ] If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
- [ ] CLA has been signed and all committers have signed the CLA in this PR.