lewtun

Support running private models with community inference API

1

As a user it would be nice to use the inference API widgets for private models generated via one of the Docker images in `api-inference-community`. Here's an example [model repo](https://huggingface.co/lewtun/superb-s3prl-osanseviero__hubert_base-asr-d570d5d0)...

I realise this is an optional dependency for `datasets` end-users, but it would be nice to include in the viewer 

Link to reproduce: https://huggingface.co/datasets/viewer/?dataset=common_voice Changing the subset to something different from `ab` seems to resolve the problem. 

Fixes small typo in dataset URL

This PR fixes a mismatch between some entries of the `human_ans_spans` column and the corresponding span of text in the `review` column. For example, line 17 of _electronics/splits/train.csv_ has in...

This issue collects links of community feedback on the type of content to include in the handbook. Feel free to post a comment below with other ideas / requests! *...

Expose a few model predictions / gold answers in the logs

1

For generative benchmarks like MATH / GSM8k / IFEval, it would be great to have some visibility in the logs on how the prompts are formatted, what the generations look...

Add Sympy equivalence for MATH / GSM8K?

1

In the [Minerva](https://arxiv.org/pdf/2206.14858.pdf) and LLeMMa papers, `sympy` is used to ensure equivalence of predicted / gold answers, e.g. ensuring $1/ \sqrt{3}$ and $\sqrt{3}/3$ are treated the same. From the Minerva...

`Could not initialize the JudgeOpenAI model` and `openi` import error

1

FYI I'm seeing a lot of these "errors" in my logs when running lighteval ``` Could not initialize the JudgeOpenAI model: [Errno 2] No such file or directory: 'src/lighteval/tasks/extended/mt_bench/judge_prompts.jsonl' ```...