mteb Add Prompt Retrieval

We could add the prompt retrieval benchmark: https://arxiv.org/abs/2209.01975

Jan 25 '23 16:01 Muennighoff

You mean add it as a task, right? along with all the datasets mentioned in the paper?

Apr 20 '24 05:04 dipam7

As a benchmark I suspect, including all its datasets.

Apr 20 '24 13:04 KennethEnevoldsen

Also cc'ing @hongjin-su here who also knows mteb quite well & may be interested in adding / helping add this

Apr 20 '24 13:04 Muennighoff

Sure, I could add this!

Apr 20 '24 15:04 hongjin-su

The performance for prompt retrieval is measured by LLM results in downstream tasks. Back then, the paper used GPT-J. Should we switch to a more up-to-date model, e.g., Llama3-8B or mistral-7B?

Apr 23 '24 03:04 hongjin-su

The performance for prompt retrieval is measured by LLM results in downstream tasks. Back then, the paper used GPT-J. Should we switch to a more up-to-date model, e.g., Llama3-8B or mistral-7B?

If those newer models mean better evaluation results, then probably a good idea to switch!

Apr 23 '24 04:04 Muennighoff

I create a pr to include 10 tasks for prompt retrieval. Feel free to check it out!

May 01 '24 04:05 hongjin-su

Just to let others contributors know that the PR (#608) was never merged. We would still welcome this submission.

Jun 05 '24 18:06 KennethEnevoldsen