Ziteng Wang

icon indicating an email [email protected]

icon company @thu-ml icon location Beijing China icon location PhD student in TSAIL Group, Department of Computer Science and Technology, Tsinghua University

Results 1 issues of Ziteng Wang

[BUG] MoE load balancing loss is accumulated twice when using activation checkpointing

3 comment

**Describe the bug** Load balancing loss is accumulated twice when using activation checkpointing **To Reproduce** Train from scratch with / without `--moe-layer-recompute`, setting `--moe-router-load-balancing-type aux_loss` **Expected behavior** Load balancing loss...

stale