mup icon indicating copy to clipboard operation
mup copied to clipboard
verified publisher icon microsoft

MuP for Mamba

Open norikazu99 opened this issue 1 year ago • 0 comments

Hello and thank you for sharing the repo. I'd like to know if muP would work out of the box with Mamba model or I would have to rescale some constants like in transformers attention_scores?

norikazu99 avatar May 30 '24 08:05 norikazu99