pygaggle icon indicating copy to clipboard operation
pygaggle copied to clipboard

[Bug?] uploaded monot5-large-msmarco-10k and monot5-large-msmarco model is exactly the same

Open soyoung97 opened this issue 1 year ago • 2 comments

Hello, I was trying to evaluate the monot5-large model on the beir benchmark, but I noticed that the uploaded models - 10k and 100k models - has exactly the same model. (when I do diff on terminal, it shows no difference) image

Also, the performance of large models is a lot lower than the performance of base models - Could you check if the two uploaded large models are accurate? (maybe relevant to the previous issue)

Thank you!

soyoung97 avatar Oct 17 '23 05:10 soyoung97

Another example showing the weight of two models is exactly the same: (code from here)

image

soyoung97 avatar Oct 18 '23 11:10 soyoung97

I ran into the same problem, I tested the 10K and 100K monoT5 on BEIR and found the results were the same, and both were very low, close to zero

KID-22 avatar Jan 03 '24 07:01 KID-22