mteb icon indicating copy to clipboard operation
mteb copied to clipboard
verified publisher icon embeddings-benchmark

Which version is used for the retrieval?

Open zhimin-z opened this issue 2 years ago • 4 comments

Unlike the reranking task of CMedQA, there is no specification of the version for this benchmark... image image Which version is used for the retrieval?

zhimin-z avatar Dec 31 '23 22:12 zhimin-z

Based on the number of samples I think CMedQAv2 is the same data as CmedqaRetrieval. Maybe @staoxiao can confirm

Muennighoff avatar Jan 01 '24 10:01 Muennighoff

Based on the number of samples I think CMedQAv2 is the same data as CmedqaRetrieval. Maybe @staoxiao can confirm

Thanks, is that fine to make its name more explicit in this case?

zhimin-z avatar Jan 01 '24 17:01 zhimin-z

Based on the number of samples I think CMedQAv2 is the same data as CmedqaRetrieval. Maybe @staoxiao can confirm

Yes, CmedqaRetrieval is v2.

staoxiao avatar Jan 02 '24 02:01 staoxiao

Based on the number of samples I think CMedQAv2 is the same data as CmedqaRetrieval. Maybe @staoxiao can confirm

Yes, CmedqaRetrieval is v2.

Thanks, can we make a pr to clearly clarify it in the names, so people won't get confused in the future? @staoxiao @Muennighoff

zhimin-z avatar Jan 02 '24 03:01 zhimin-z

I have added a specification in the dataset description in #886

KennethEnevoldsen avatar Jun 05 '24 18:06 KennethEnevoldsen