TypeError: pass_k int not iterable in evaluate.py

[KedarnathKC]

Hi,

Here is the command I am using: bigcodebench.evaluate --execution local --split complete --subset full --samples /scratch3/workspace/wenlongzhao_umass_edu-reason/dev_kedar/Small-LLM-Reasoning/scratch/notebooks/output.jsonl --pass_k "1"

I was trying to locally evaluate the results, and I used --pass_k "1", it throws a type error as below: TypeError: 'int' object is not iterable at line 367 in evaluate.py:

I see that at line 219, when pass_k is not a string, you assign it as is to passk.

Later, when updating pass_at_k, you iterate over passk, which is causing this issue.

When I debugged and tried to see what was happening, when Fire parses this parameter, it considers pass_k as an integer, which is causing this issue.

✅ Suggested Fix Explicitly normalize pass_k to a list of integers, like:

if isinstance(pass_k, str):
    passk = [int(k) for k in pass_k.split(",") if k.strip()]
elif isinstance(pass_k, int):
    passk = [pass_k]
elif isinstance(pass_k, (list, tuple)):
    passk = list(pass_k)
else:
    raise ValueError(f"Invalid type for pass_k: {type(pass_k)}")

Happy to fix this and open a pull request if this sounds good to the maintainers.

[terryyz]

Hi @KedarnathKC, thanks for reporting this issue! It'd be great if you can submit a PR :-)

[KedarnathKC]

@terryyz I have fixed and have raised a PR for the same.