Open-Llama icon indicating copy to clipboard operation
Open-Llama copied to clipboard

Published 20 hours ago • verified publisher icon s-JoL

RoPE bias

Open JaheimLee opened this issue 1 year ago • 4 comments

llama应该默认没有启用bias项。但按照苏神最新思路，把q,k的bias项加回来可以明显提升长度外推性能，作者考虑预训练测试一下不 https://kexue.fm/archives/9577

JaheimLee avatar Apr 11 '23 05:04 JaheimLee

赞，再预训练的时候我会加上

s-JoL avatar Apr 11 '23 06:04 s-JoL

请问目前公开的代码预训练的时候加上bias项了吗

chensongcan avatar Apr 24 '23 07:04 chensongcan

目前加上bias项的还在训练中，将在训练好之后再release出相应代码

s-JoL avatar Apr 24 '23 09:04 s-JoL

这个项目是基于llama从0开始训练是吗

chensongcan avatar Apr 24 '23 09:04 chensongcan