magicoder
magicoder copied to clipboard
The templates used in reproducing the eval results: why adding the instruction again after "### Response: "?
There is an input format mismatch between the eval and training process. Do you intend to emphasize the problem before the model generates its output?
When doing the Humaneval(+) eval, the compiled inputs are as follows, eg.:
@@ Instruction
Write a solution to the following problem:
```python
def fib(n: int):
"""Return n-th Fibonacci number.
>>> fib(10)
55
>>> fib(1)
1
>>> fib(8)
21
"""
@@ Response
def fib(n: int):
"""Return n-th Fibonacci number.
>>> fib(10)
55
>>> fib(1)
1
>>> fib(8)
21
"""```
But in the code files about data processing and training, the instruction data would be compiled as:
You are an exceptionally intelligent coding assistant that consistently delivers accurate and reliable responses to user instructions.
@@ Instruction Write a solution to the following coding problem: {problem}
@@ Response {response}
There is no such **_rephrasing/emphasizing_** in the training data of Magicoder.
From the eval results this mismatch seems not to bring obvious negative effects, but did you deliberately do so?