gritlm

gritlm copied to clipboard

Can I select causal attention for retrieval embeddings when using GritLM

17

In the paper, the ablation study about attention emb and gen is interesting. Are these models all different models using each attention? Can I select causal attention for both cases...

Yangseung

How do I get the sentence embedding from GritLM/emb_m7_nodes16_fast?

4

Continuing from our conversation in https://github.com/ContextualAI/gritlm/issues/13 I just think it needed a new ticket at this point. I am trying to finetune embeddings only so I took your(@Muennighoff 's) recommendation...

phartman-keysight

Add support for encoding pretokenized sequences

4

Useful for batch processing and making embeddings cache of numerous documents with dataloaders. The results for dict and the vanilla strings list are identical, although for the raw tokenized 'transformers'...

kabachuha

When training a unified model, TypeError: MistralForCausalLM.forward() got an unexpected keyword argument 'is_causal'

5

Hello! I meet a problem when I train the model in the unified mode. First, I would like to share that when I **evaluate** several models in the artifacts (for...

zillion-zhao

maharshi95

projection layer

2

if i use projection layer for ddp it will cause: RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1)...

CementMaker

how to add projection

1

How to add a projection module to an existing model? The hidden_state of this model is too large. i added projection=128 in the train_embonly.sh, but the result files don't not...

threestone965

Can this training framework be used for training gte-qwen2-7b-instruct model？

1

Hello. I wanted to use gritlm to a open-source embedding model —— gte-qwen2-7b-instruct, but I encountered some problems: ``` [rank1]: Traceback (most recent call last): [rank1]: File "/code/xx/LLM_mine/recall/reference/gritlm/gritlm/training/run.py", line 438,...

Double-bear

save checkpoint with error

1

set max_steps=500, save_steps=100 When it reaches step 100, the checkpoint is saved successfully but nccl_timeout is displayed

13613979212

Hi, while reviewing the licenses for this repository and the model it depends on, I noticed a potential inconsistency that could cause confusion or legal risks in some situations. Your...

yueyangchen1