Terry Yue Zhuo

Sure! My email address is `[email protected]`.

Hi, I'm working (slowly) on a new version on the [dev](https://github.com/terryyz/llm-code-eval/tree/dev) branch, which supports the open model inference. I previously tested with LLaMA only and haven't had time to compute...

Hi @gvijqb, No problem! Please let me know if you'd like to collaborate on this project and beyond :) Cheers, Terry

Not sure if MonsterAPI may support some computational resources. I'm a bit short of good GPUs these days 😞

Hi @littlepopy, the details were discussed in Appendix B.2 in the [paper](https://arxiv.org/pdf/2302.05527.pdf), where the data was collected from [HumanEval-X](https://huggingface.co/datasets/THUDM/humaneval-x). Besides, you may also find [our paper](https://arxiv.org/abs/2304.14317) useful. [This folder](https://github.com/terryyz/llm-code-eval/tree/main/data) may...

Thanks @dmelcer9! It makes sense :) We didn't think about this when developing the initial tasks. We will incorporate this change in the next dataset release.

Thanks for asking! I'm quite busy these two months, but I might be able to take a look in October if no one else has the bandwidth to add the...

@hvaara Let me know how it goes with the model verification :) If it works, please put it into `tools/fix_0.2.1` and point to the data to `v0.1.3`.

Thanks @hvaara! Could you move the part to a file called `fix_v023.py`, which will be a patch based on the v0.2.3 release.

Hi @s-smits, Thanks for the request! I'll try to support customized prompts in the next version. For now, please: 1. Clone the repo 2. Modify the [instruction prompt](https://github.com/bigcode-project/bigcodebench/blob/main/bigcodebench/generate.py#L161). You don't...