human-eval
human-eval copied to clipboard
openai
evaluate_functional_correctness can't run
I created a conda environment with python3.7 using the exact same command in the doc. Then, I used openai's text-davinci-002 to generate a samples.jsonl file with 3 results for each problem.
Calling evaluate_functional_correctness samples.jsonl, I got the error message as below. I also tried to evaluate the example results with evaluate_functional_correctness data/example_samples.jsonl --problem_file=data/example_problem.jsonl, and got the same error.
I wonder how to fix it?
Error message:
Reading samples...
6it [00:00, 7427.93it/s]
Running test suites...
0%| | 0/6 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/opt/miniconda3/envs/codex/bin/evaluate_functional_correctness", line 33, in
If you are on windows, this problem can be solved by declaring a global var of unsafe_execute just above the function.
And after this you would encounter Attribute error for unsafe_execute, that can be solved by installing multiprocess library and then replacing "multiprocessing" with multiprocess instead.
Another error on windows would be of pass@=0 instead of 0.4999, this can be solved by removing the timeout function in execution.py for sample sanity check only, for testing on generated samples you have to use timeout but have to find a way to use something different than setitimer() function which causes the pass@=0 issue.
reinstall multiprocess, and replace multiprocessing used in human_eval/execution.py， then here we go.
when I run on mac,I get the same error,can anyone help?thanks a lot
so sad!!! I also have the same problem. Do u solve it, my friend? thx
After importing and change from multiprocessing to multiprocess. What worked for me was to use this
windows machine:
import threading
class TimeoutException(Exception):
pass
@contextlib.contextmanager
def time_limit(seconds: float):
timer = threading.Timer(seconds, lambda: (_ for _ in ()).throw(TimeoutException("Timed out!")))
timer.start()
try:
yield
finally:
timer.cancel()
linux machine
@contextlib.contextmanager
def time_limit(seconds: float):
def signal_handler(signum, frame):
raise TimeoutException("Timed out!")
signal.setitimer(signal.ITIMER_REAL, seconds)
signal.signal(signal.SIGALRM, signal_handler)
try:
yield
finally:
signal.setitimer(signal.ITIMER_REAL, 0)
pass
On a Mac with Python 3.10 installing multiprocess and removing imports and usages of multiprocessing worked for me.
After importing and change from multiprocessing to multiprocess. What worked for me was to use this
windows machine:
import threading class TimeoutException(Exception): pass @contextlib.contextmanager def time_limit(seconds: float): timer = threading.Timer(seconds, lambda: (_ for _ in ()).throw(TimeoutException("Timed out!"))) timer.start() try: yield finally: timer.cancel()linux machine
@contextlib.contextmanager def time_limit(seconds: float): def signal_handler(signum, frame): raise TimeoutException("Timed out!") signal.setitimer(signal.ITIMER_REAL, seconds) signal.signal(signal.SIGALRM, signal_handler) try: yield finally: signal.setitimer(signal.ITIMER_REAL, 0) pass
For Windows user, here is more things need to be done:
- add
if __name__ == "__main__":before thesys.exit(main())otherwise you will get a error which recommends you to use "freeze_support()" - install multiprocess and removing imports and usages of
multiprocessingor you will get "Ran out of input" error. Please ignore the IDE's "can't find **()" warning as the package multiprocess doesn't announce them in init.py but it still works. - change the time_limit function is necessary as the origin one didn't work properly.
- TimeoutException class defination is optional
reinstall multiprocess, and replace multiprocessing used in human_eval/execution.py， then here we go.
This works well for me on windows 11. Thank you so much!
