Niklas
Fixes https://github.com/allenai/OLMo/issues/457 As this is a breaking change, it probably makes sense to release a new version with it
### 🐛 Describe the bug In the HF code we use OLMo but in training it's Olmo - This creates some inconsistencies when importing from the training modeling file ----...
Would be great if the [HFDataLoader](https://github.com/beir-cellar/beir/blob/c3334fd5b336dba03c5e3e605a82fcfb1bdf667d/beir/datasets/data_loader_hf.py#L89) would support dataset revisions. Also imo it would be much better to merge the qrels HF repos with the regular ones & load the...
Hi, great work! Why are you feeding the instruction both into the encoder & decoder? I would have expected only the encoder to receive the instruction, however, for HumanEval you...
I would like to get all [xP3](https://huggingface.co/datasets/bigscience/xP3) prompts merged into eval-hackathon & then have eval-hackathon be merged into main once & for all, but I'm not sure if you want...
We could add the prompt retrieval benchmark: https://arxiv.org/abs/2209.01975
Would be a cool addition together with MMTEB although a bit orthogonal to the multilingual aspect; cc @amanpreet692