mteb icon indicating copy to clipboard operation
mteb copied to clipboard

Problems with mteb_meta for german evaluation

Open achibb opened this issue 8 months ago • 6 comments

Hi everyone, I am having troubles with generating the mteb_meta for German, with just running the script.

I am currently trying to format results but it does not seem to work straight away with "mteb_meta.py" - any idea? I just get a blank metadata file:

tags:

mteb model-index: name: gbert-large

results:

it gives this for every dataset:

WARNING:mteb.evaluation.MTEB:Passing task names as strings is deprecated and will be removed in the next release. Please use tasks = mteb.get_tasks(tasks=[...]) method to get tasks instead.
INFO:main:Skipping AmazonCounterfactualClassification as split test not present.
WARNING:mteb.evaluation.MTEB:Passing task names as strings is deprecated and will be removed in the next release. Please use tasks = mteb.get_tasks(tasks=[...]) method to get tasks instead.
INFO:main:Skipping AmazonReviewsClassification as split test not present.

Do I need to modify something on the code?

achibb avatar May 29 '24 14:05 achibb