I added HaluEval to lm-eval-harness, can you please double check?

[pminervini]

trafficstars

Here is my pull request: https://github.com/EleutherAI/lm-evaluation-harness/pull/1076

Thanks!

[Xiaoxue-xx]

Sorry for the late reply. We have double-checked the PR and think there is no problem with this implementation.

[pminervini]

@Xiaoxue-xx thank you! 🙂 You can find the latest version of the tasks here: https://huggingface.co/spaces/hallucinations-leaderboard/leaderboard/tree/main/src/backend/tasks/halueval We used it for our Hallucinations Leaderboard: https://huggingface.co/blog/leaderboards-on-the-hub-hallucinations

@Xiaoxue-xx, would you also be happy if we switch from an open-ended generation task to a multiple-choice task, where the model is only allowed to generate "yes" or "no"?

[Xiaoxue-xx]

Sure. Thank you for your attention to HaluEval！