Xiaosong,He

能给一份S^2 Attension推理的代码吗？

4

RT

does anybody try to uptrain an exsiting model to obtain a gqa-version model?

2

I'am trying to do this with my own implementation (not with T5), but the initial loss is pretty high (10+), I am not sure whether this is normal.

[Bug]: The final reason why you will get a model that cannot stop generation when you fine-tune the Qwen2.5-7b-base use Lora and a non-<|endoftext|> token as eos_token.

9

### Model Series Qwen2.5 ### What are the models used? Qwen2.5-7b-base ### What is the scenario where the problem happened? sft with huggingface trainer ### Is this a known issue?...