wenet icon indicating copy to clipboard operation
wenet copied to clipboard
verified publisher icon wenet-e2e

[transformer] add qk norm

Open Mddct opened this issue 1 year ago • 0 comments

多模态和部分大模型中采用qk norm 来稳定训练 （apple dmel中也用到了）(有益于bestrq 训练 和稳定梯度) 截屏2024-08-01 13 31 22

TODO:

  • [ ] conformer result

Mddct avatar Aug 01 '24 05:08 Mddct