mteb
mteb copied to clipboard
embeddings-benchmark
CodeSearchNet task
would the maintainers be interested in the addition of a code retrieval task (CodeSearchNet, uses text queries to retrieve code documents), either as a new code retrieval type or added into the existing retrieval category?
would the maintainers be interested in the addition of a code retrieval task (CodeSearchNet, uses text queries to retrieve code documents), either as a new code retrieval type or added into the existing retrieval category?
Yes definitely! I think it would be best to add it to the existing retrieval category if possible. It would require some changes in the code to differentiate between BEIR and non-BEIR, but should not be too difficult.
The language splits would then be the coding languages go, java etc.
Do you want to tackle this? Very happy to help along the way! 🤗
i haven't gotten around to this yet, but would be happy to give it a shot
Amazing! Let me know when you run into problems 👍
I think @bwanglzu & team have also been working on this - Let us know if we can help in any way!
hi @Muennighoff, @Manouchehri and all, yes we're working on this. Maybe a bit more, we'll add ~3 coding tasks to MTEB :)
reason behind this is we're training a coding embedding model, similar as: jina-embeddings-v2 :)
hi @Muennighoff, @Manouchehri and all, yes we're working on this. Maybe a bit more, we'll add ~3 coding tasks to MTEB :)
reason behind this is we're training a coding embedding model, similar as: jina-embeddings-v2 :)
Amazing really looking forward to it! Let us know if we can help :)
