Added ArXiv Hierarchical clustering (S2S and P2P)

[x-tabdeveloping]

Checklist for adding MMTEB dataset

Reason for dataset addition: Changed ArXiv clustering tasks (S2S and P2P) to hierarchical (two levels separated by dots). #696

• [x] I have tested that the dataset runs with the mteb package.
• [ ] I have run the following models on the task (adding the results to the pr). These can be run using the mteb run -m {model_name} -t {task_name} command.
  • [ ] sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • [ ] intfloat/multilingual-e5-small
• [ ] I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• [x] If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
• [x] I have filled out the metadata object in the dataset file (find documentation on it here).
• [x] Run tests locally to make sure nothing is broken using make test.
• [x] Run the formatter to format the code using make lint.
• [ ] I have added points for my submission to the points folder using the PR number as the filename (e.g. 438.jsonl).

[x-tabdeveloping]

I can't run it cause I don't have compute access today, will run when I can.

[x-tabdeveloping]

Also we should probably wait for the other PR (#694 ) to merge, as the stratified subsampling will fail without the new code.

[Muennighoff]

This will change the scores of ArxivClustering, correct? I think we need to be more careful with maintaining backward compat., e.g. this point here: https://github.com/embeddings-benchmark/mteb/pull/481#issuecomment-2096636221 Maybe we can allow to also evaluate with the previous way.

[x-tabdeveloping]

Yeah, we need to be careful about this! Can't we use a previous release of MTEB for evaluating models for the current leaderboard? I think it would be really nice if we wouldn't have to worry about production use while trying to get the MMTEB project and paper done as fast as we can. How should we go about this @Muennighoff ?

[Muennighoff]

Yeah, we need to be careful about this! Can't we use a previous release of MTEB for evaluating models for the current leaderboard? I think it would be really nice if we wouldn't have to worry about production use while trying to get the MMTEB project and paper done as fast as we can. How should we go about this @Muennighoff ?

Actually, I think it is fine because the original ArxivClustering is still supported right? It is just superseded by this one.

Does this mean that when people run https://github.com/embeddings-benchmark/mteb/blob/main/scripts/run_mteb_english.py it is still the same datasets as originally in MTEB? To select the newer versions, do people need to set the name as e.g. ArxivClusteringP2P.v3, correct? @KennethEnevoldsen

[KennethEnevoldsen]

@x-tabdeveloping we should get this PR finished up

[x-tabdeveloping]

Okay, I will just go for dummy subsampling here as in VG, and we can add stratified subsampling, once it is properly addressed.