FlagEmbedding
FlagEmbedding copied to clipboard
FlagOpen
Retrieval and Retrieval-augmented LLMs
想请问一下我想进行蒸馏微调reranker，如何构造对应的微调数据集，获得相应的 teacher_scores 。我打算利用大模型得到这个分数，构造出来的分数需要满足什么要求？必须进行归一化后在 【0-1】吗？
祝贺你们做出了如此杰出的工作，我对这篇论文很感兴趣，在阅读论文之后有一些问题不知道能否解答？ 1. 在self-attn模块中，由于第二个chunk依赖于第一个chunk生成的beacon，这是否会导致无法并行**训练**？即是否需要按顺序依次计算每个chunk？而不能像传统架构中那样并行计算？这似乎使得训练时，每个attention模块的计算延迟与chunk数量相关？当然这在推理解码过程中是没问题的。 2. 在插入beacon之后，计算第二个chunk时， $\langle b \rangle^2_1$ 的存在是否会导致影响 $x^2_2$ 与 $x^2_3$ 的相对位置（增加了1）？另外 $\langle b \rangle^1_1$ 与 $\langle b \rangle^1_2$ 的位置关系在第一个chunk和第二个chunk计算过程中的位置关系似乎也改变了？不知道这种细节是否会对模型性能有影响？
Hi, there is an error of importing packages in my code. I did all the steps included: ```bash git clone https://github.com/FlagOpen/FlagEmbedding.git cd FlagEmbedding/research/visual_bge pip install -e . ``` and ```bash...
Hi @staoxiao I see you guys have code for fine-tuning Gemma-2 as a reranker. Can this code be used for other LLMs like LLaMA or Mistral?
# packages in environment at /root/miniconda3/envs/qwen3flag: # # Name Version Build Channel datasets 2.19.0 pyhd8ed1ab_0 conda-forge pyarrow 15.0.2 py310h08f37a1_55_cpu conda-forge 有没有人知道为啥 Collecting pyarrow>=12.0.0 (from datasets>=2.19.0->FlagEmbedding==1.3.5) Using cached pyarrow-21.0.0.tar.gz (1.1 MB)...
> uv pip install 'FlagEmbedding[finetune]' Resolved 95 packages in 31ms × Failed to build `flash-attn==2.8.2` ├─▶ The build backend returned an error ╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit status: 1)...
Hi @staoxiao @hanhainebula, You state in your paper, 'In the initial phase, we employed approximately 6000 steps to perform warm-up on dense embedding, sparse embedding and multi-vectors. Subsequently, we conducted...
Hi @staoxiao @hanhainebula, Your paper doesn't seem to mention whether you sampled examples in each batch from the dataset or whether you allowed each batch to contain samples from different...
`[HAMI-core Msg(48605:139901809241280:libvgpu.c:836)]: Initializing..... [HAMI-core Msg(48684:140081705012416:libvgpu.c:836)]: Initializing..... [HAMI-core Warn(48684:140081705012416:utils.c:183)]: get default cuda from (null) [HAMI-core Msg(48684:140081705012416:libvgpu.c:855)]: Initialized [2025-06-20 01:28:10,794] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2025-06-20 01:28:12,061] [INFO] [logging.py:107:log_dist]...
