wyooyw

MultiSTGCnet 18182672 WangYiou

Fix expert grad scaling problem with ZeRO optimizer

3

Fix [#6545] work: - expert gradient average: divide edp_world_size -> divide dp_world_size - unit test: make sure model with different dp/ep has same expert gradient

[BUG] Expert gradient scaling problem with ZeRO optimizer

1

**Describe the bug** When using ZeRO optimizer training MoE model, the gradient of the expert weights is **ep_size times larger than** the true gradient. **Related issue & pr** Issue [#5618]...