zou kai issues

Repositories
Issues
Comments

Results 1 issues of


                                            zou kai

Enable compatibility of sparse MOE training under DeepSpeed ZeRO3

According to https://github.com/microsoft/DeepSpeed/pull/4966, ZeRO3 in DeepSpeed does not work with MoE models because the order of executing modules can change at every forward/backward pass and a new API is implemented...