bigcodebench icon indicating copy to clipboard operation
bigcodebench copied to clipboard

What is the prompt format?

Open ccocks opened this issue 7 months ago • 9 comments
trafficstars

I have an OpenAI compatible endpoint that I'm prepping up for evaluation, and I want to know what is the FINAL end request being sent? e.g: (I'm looking at the HF dataset https://huggingface.co/datasets/bigcode/bigcodebench-hard/viewer/default/v0.1.0_hf?row=0&views%5B%5D=v010_hf)

[
{
"role":"user"
"content": "{complete_prompt if complete else instruct_prompt}"
},
{
"role":"assistant"
"content": "Sure, I can do that! Blah Blah Blah: ```python {canonical_solution} ``` This function does this and that..."
}
]

Basically what format should I expect and what should I send back?

ccocks avatar Apr 01 '25 16:04 ccocks