HumanEval/84, adding initial prompt produces irrelevant code.

[SmartManoj]

initial_prompt = '"Don\'t add any test functions. Complete this function based on the docstring."\n\n'

MRE:

vllm serve Qwen/Qwen2.5-Coder-32B-Instruct

import openai

model = "Qwen/Qwen2.5-Coder-32B-Instruct"
openai.api_key = 'test'
openai.base_url = base_url

def completion(prompt):
    response = openai.completions.create(
        model=model,
        prompt=prompt,
        seed=42,
        temperature=0.0,
        max_tokens=1000,
        stop=["\n\n", '```'],
    )
    return response.choices[0].text


initial_prompt = '"Don\'t add any test functions. Complete this function based on the docstring."\n\n'
prompt = '''
def solve(N):
    """Given a positive integer N, return the total sum of its digits in binary.
    
    Example
        For N = 1000, the sum of digits will be 1 the output should be "1".
        For N = 150, the sum of digits will be 6 the output should be "110".
        For N = 147, the sum of digits will be 12 the output should be "1100".
    
    Variables:
        @N integer
             Constraints: 0 ≤ N ≤ 10000.
    Output:
         a string of binary number
    """
'''    
print(completion(prompt))
print('---')
print(completion(initial_prompt + prompt))

Output: Without initial prompt

    # Calculate the sum of digits
    digit_sum = sum(int(digit) for digit in str(N))
    # Convert the sum to binary and return
    return bin(digit_sum)[2:]

With initial prompt

    # Convert the integer to binary and remove the '0b' prefix
    binary_representation = bin(N)[2:]
    
    # Calculate the sum of the digits in the binary representation
    digit_sum = sum(int(digit) for digit in binary_representation)
    
    # Convert the sum to a binary string
    binary_sum = bin(digit_sum)[2:]
    
    return binary_sum

[jamesmurdza]

What is the issue with the output code?

[SmartManoj]

# Convert the integer to binary and remove the '0b' prefix binary_representation = bin(N)[2:]

It converted the input also to binary.

[cyente]

different prompts may lead to different answers, we are still working on it .