lm-evaluation-harness icon indicating copy to clipboard operation
lm-evaluation-harness copied to clipboard

Published 20 hours ago • verified publisher icon EleutherAI

Premature `num_fewshot` check with `fewshot_as_multiturn`

Open baberabb opened this issue 6 months ago • 0 comments

main raises a ValueError if fewshot_as_multiturn and no num_fewshot. However, num_fewshot can also be set in the task YAML, which is processed later. Think we should remove this especially because num_fewshot sets the global fewshots ignoring task specific values.

https://github.com/EleutherAI/lm-evaluation-harness/blob/b70af4f51481db15ec3068d26cde3d76dd0201e7/lm_eval/main.py#L295-L300

baberabb avatar Jul 30 '24 20:07 baberabb