langtest icon indicating copy to clipboard operation
langtest copied to clipboard

Published 20 hours ago • verified publisher icon JohnSnowLabs

Preparing Embeddings Benchmarks (LangTest)

Open ArshaanNazir opened this issue 1 year ago • 9 comments

ArshaanNazir avatar Dec 27 '23 17:12 ArshaanNazir

@ArshaanNazir please add your self as an assignee.

JustHeroo avatar Dec 28 '23 08:12 JustHeroo

@ArshaanNazir could you share your updates?

JustHeroo avatar Feb 27 '24 10:02 JustHeroo

@chakravarthik27 please share the latest updates.

JustHeroo avatar Mar 05 '24 10:03 JustHeroo

Hi @JustHeroo,

We have finished benchmarking the paul_graham dataset and now I am working on creating curated datasets for retrieval evaluation to do the benchmarking embedding models. I plan to generate question-answer pairs for each dataset and implement metrics to evaluate embedding models.

chakravarthik27 avatar Mar 05 '24 17:03 chakravarthik27

@chakravarthik27 please update the latest status here.

JustHeroo avatar Mar 11 '24 19:03 JustHeroo

@ArshaanNazir could you share your update

Cabir40 avatar Mar 12 '24 10:03 Cabir40

@Cabir40 Kalyan is working on it. He will be enhancing embedding benchmarks for other retrieval tasks. He is working on FinBERT-QA right now.

ArshaanNazir avatar Mar 12 '24 10:03 ArshaanNazir

@chakravarthik27 is there any update?

Cabir40 avatar Mar 19 '24 09:03 Cabir40

Hi @Cabir40,

Still, it is in progress, I am currently working alone on the langtest project and implementing a high priority feature similar to Open LLM Leaderboard by Hugging Face(Eleuther). It may take some time to complete the embedding benchmarks.

chakravarthik27 avatar Mar 19 '24 10:03 chakravarthik27