FlagEmbedding icon indicating copy to clipboard operation
FlagEmbedding copied to clipboard

something is wrong with hard neg mine

Open QuangTQV opened this issue 1 year ago • 2 comments

I tried hard neg mining, but when running on 2 different gpu's (namely t4 and a100), t4 only took a few seconds while a100 took 20 minutes, and their hard neg results were also completely different.

QuangTQV avatar May 13 '24 23:05 QuangTQV

I think this error is caused by faiss. You can use faiss on CPU instead of GPU

staoxiao avatar May 14 '24 04:05 staoxiao

I think this error is caused by faiss. You can use faiss on CPU instead of GPU

My data has 7000 queries and 17 keys, how long should it take to mine hard negatives (because I still don't know if running in 10s or running in 20 minutes is wrong)

QuangTQV avatar May 14 '24 07:05 QuangTQV