DeepSpeedExamples
DeepSpeedExamples copied to clipboard
Enable overlap_comm for better performance
Enable overlap of backward computation and gradient all-reduce. This produces 1.05x end-to-end speedup in SFT training with my settings. See also https://github.com/microsoft/DeepSpeed/pull/4887.