Oleg Filatov

icon indicating a website link linkedin.com/in/oleg-filatov/

icon location Cologne, Germany icon location LLM researcher // Helmholtz AI & Jülich Supercomputing Centre

Results 1 issues of Oleg Filatov

_rescale_parameters() inconsistent with the paper for the tied embedding scenario?

2 comment

Hi! I've been looking into the integration of `muP` into the `Megatron-LM` setup and I was wondering about the `_rescale_parameters()` method of `MuReadout` in case of shared (tied) input/output embeddings....