DeepSpeed icon indicating copy to clipboard operation
DeepSpeed copied to clipboard
verified publisher icon microsoft

[Bug] - summing over DP instead of MP

Open EricSteinberger opened this issue 3 years ago • 0 comments

Hi Deepspeed team!

Should this be summing over the MP group rather than the DP group? The comment says MP and the code does DP.

https://github.com/microsoft/DeepSpeed/blob/b3388e1418e80141a902886dec39127b851a1f0a/deepspeed/runtime/zero/stage_1_and_2.py#L1156

EricSteinberger avatar Jul 14 '22 02:07 EricSteinberger