Oleg Filatov

icon indicating a website link linkedin.com/in/oleg-filatov/

icon location Cologne, Germany icon location LLM researcher // Helmholtz AI & Jülich Supercomputing Centre

Results 3 comments of Oleg Filatov

The plot got from muP coord_check seems not horizontal, which may indicates there exits a bug in the muP implementation?

I was looking into the muP implementation in `gpt-neox` to contrast it with the `Megatron-LM` setup and accidentally found this issue :) I am thinking, could LR schedule be the...

The plot got from muP coord_check seems not horizontal, which may indicates there exits a bug in the muP implementation?

@marcobellagente93 Oh yes, now it's indeed nicely flat curves, great ! :)

Increasing coord check for the network output

@AkshitaB (very delayed reply but still might be helpful) From my experience, I also tried query/readout zero-init and it didn't help. However, what I saw is that while growing at...