FlagEmbedding icon indicating copy to clipboard operation
FlagEmbedding copied to clipboard

Error occurred in hard negative examples mining

Open LLLLucensus opened this issue 1 year ago • 1 comments

已安装faiss-gpu-cu11,遇到以下报错何解?感谢~~ 机器:4*v100 16G 运行命令: python hn_mine.py
--input_file toy_finetune_data.jsonl
--output_file toy_finetune_data_minedHN.jsonl
--range_for_sampling 5-8
--negative_number 2
--use_gpu_for_searching
--embedder_name_or_path ../../BAAI/bge-m3

inferencing embedding for corpus (number=80)-------------- initial target device: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:13<00:00, 3.31s/it] Chunks: 0%| | 0/4 [00:00<?, ?it/s] Chunks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:13<00:00, 3.31s/it] inferencing embedding for queries (number=10)-------------- Chunks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 67.19it/s] create index and search------------------ Traceback (most recent call last): File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 233, in main(data_args, model_args) File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 214, in main find_knn_neg( File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 164, in find_knn_neg index = create_index(p_vecs, use_gpu=use_gpu) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 96, in create_index index = faiss.IndexFlatIP(len(embeddings[0])) ~~~~~~~~~~^^^ KeyError: 0

LLLLucensus avatar Jan 17 '25 08:01 LLLLucensus

你好,@LLLLucensus。感谢指出这个 bug,现已修复:https://github.com/FlagOpen/FlagEmbedding/pull/1340

hanhainebula avatar Jan 18 '25 06:01 hanhainebula