Terry Yue Zhuo

Instruction_prefix -> [these two variables](https://github.com/bigcode-project/bigcodebench/blob/main/bigcodebench/generate.py#L150-L151) The "response" variable is used for profiling, the same design as inside EvalPlus. It won't be used by any model APIs.

The pointer of Instruction_prefix has been fixed. > Is there a way I can debug this? I think you may use this on Windows, where you are supposed to run...

`AttributeError: module 'os' has no attribute 'killpg'` is a part of the evaluation logic. If you fail on the os.killpg`, you basically cannot run the evaluation correctly. You have to...

Sorry, missed this. I'm not sure why you are evaluating without using any of the provided execution environment..? For example, you can freely use the [Gradio Endpoint](https://github.com/bigcode-project/bigcodebench#-remote-evaluation) on the [Hugging...

Thanks for getting back to me @ArthurZucker! The extra spaces usually appear in the prompt part of the model outputs. Here is an example of `01-ai/Yi-1.5-9B-Chat`: ```python {"task_id": "BigCodeBench/0", "solution":...

It's on https://github.com/bigcode-project/bigcodebench/blob/bbe93d673fd236e99b81cd2d7f110b63c9c2da35/bigcodebench/model.py#L137 and https://github.com/bigcode-project/bigcodebench/blob/bbe93d673fd236e99b81cd2d7f110b63c9c2da35/bigcodebench/model.py#L197. `self.tokenizer = AutoTokenizer.from_pretrained(self.tokenizer_name, **kwargs)` Let me try `add_prefix_space=False` first.

> The issue could just as well be the chat template call, given that this is something that was touched, while the `tokenizer.json` and `tokenizer_config` where not in that timeframe...

~`add_prefix_space=False`: extra space exists.`legacy=False, from_slow=True`: extra space exists.`AutoTokenizer.from_pretrained(name, add_prefix_space=False, legacy=False, from_slow=True, **kwargs)` also won't work.version: `v4.42.3` 😢~ My bad, the extra space no longer exists. I'll check the final results...

@ArthurZucker BTW, if `add_prefix_space=False` fixes the performance degradation, is it still considered a bug for the tokenizer? I mean, I didn't expect to add `add_prefix_space=False` for example 🤔

Both `add_prefix_space=False` and `legacy=False` work for `transformers` only, but not `vllm`. I'll run the full generation with `legacy=False`, which takes a bit of time to run with the pure `transformers`.