human-eval icon indicating copy to clipboard operation
human-eval copied to clipboard

Published 20 hours ago • verified publisher icon openai

bug in estimate_pass_at_k

Open sidaw opened this issue 1 year ago • 0 comments

https://github.com/openai/human-eval/blob/312c5e5532f0e0470bf47f77a6243e02a61da530/human_eval/evaluation.py#L26

This code returns 1 when c=0 and n < k, whereas 0 is expected.

sidaw avatar Jan 18 '24 23:01 sidaw