Running run_eval.py

[chenzimin]

Thanks for script for evaluation on the CodeNet dataset!

I am trying to evaluate some predictions on the dataset with the command:

python3 src/codenet_eval/run_eval.py --eval_config eval_files/example_eval_config.yaml

In the output report file, all input_* and generated_answers_* columns are either null or 0. I tried to submit the file (generated from by setting the temp_dir option to run_eval.py) to the AtCoder website and it was able to compile and run.

In attachment I have uploaded one line of the jsonl files and yaml config file. Thanks if you can take a look at this.

example_data.zip

[chenzimin]

I found out the reason to be that it is because the default value of return_if_acc_below and cflags in the config file pie-perf/src/codenet_eval /evalconfig.py are default to None.

    return_if_acc_below: float = None #  type: ignore | if the accuracy is below this value, then return immediately
    cpp_results_path: str = None #  type: ignore | the path to the results of compiling and running the ref cpp code
    cflags: str = None #  type: ignore | the cflags to use for compiling the code

This causes the split call in cflags.replace('"', "") from pie-perf/src/codenet_eval /sandbox.py to be called on a None object.

Similarly for return_if_acc_below, when default to None, it will raise a exception saying that None cannot be compared to float, also in the pie-perf/src/codenet_eval /sandbox.py file.