mteb
mteb copied to clipboard
embeddings-benchmark
Autogenerate table of tasks
Seems like the table of tasks can be autogenerated from the existing metadata.
I would suggest removing it from the readme and then linking to a new autogenerated markdown.
What are some thoughts on this?
In the same vein, I'd love if the metadata was a dataclass as well. This makes it easy to:
- Type-hint the attributes
- Auto-complete in IDEs
- Get runtime-errors if keys/attributes are missing
I've created this as a separate issue, #257.
Seems like the table of tasks can be autogenerated from the existing metadata.
I would suggest removing it from the readme and then linking to a new autogenerated markdown.
What are some thoughts on this?
Yep let's go! Autogeneration >>> manual generation 🚀
Wonderful glad we agree! Will see if I can find the time to implement it
This PR awaits #260. @MartinBernstorff Since the table contains the size of the datasets and it would be problematic to fetch all datasets to generate it, it might be ideal to the dataset sizes to the metadata. What do you think?
Makes a lot of sense. There might be some work in formalising what "size" means for each task type, but let's cross that bridge when we get there!
