lighteval icon indicating copy to clipboard operation
lighteval copied to clipboard

`Could not initialize the JudgeOpenAI model` and `openi` import error

Open lewtun opened this issue 4 months ago • 1 comments

FYI I'm seeing a lot of these "errors" in my logs when running lighteval

Could not initialize the JudgeOpenAI model:
[Errno 2] No such file or directory: 'src/lighteval/tasks/extended/mt_bench/judge_prompts.jsonl'

This doesn't cause an exception, but maybe good to clean up from the logs?

I also learned that you need the openai dep even if you don't run mt_bench due to an import error from this line: https://github.com/huggingface/lighteval/blob/fc428f600c40b214b4d784540d1a0db5ed193d30/src/lighteval/metrics/llm_as_judge.py#L30

I would recommend either promoting the openai dep to a core dep or having a checker like is_openai_available() then only importing the method if it is

lewtun avatar Apr 22 '24 09:04 lewtun