Auto scaling factor tuning for FP8 collective communication

[tocean]

What would you like to be added: Tune scaling factor automatically for fp8 collective communication.

Why is this needed: Reduce the scaling factor to min value across all GPUs may cause underflow. What's more, the gradient reduced may not be the real average gradient.

Without this feature, how does current msamp work： In FP8Linear's backward, it synchronizes scaling factor's minimum value to all GPUs. And in optimizer's all_reduce_grads, it will synchronizes average gradient to all GPUs. It will first compute the sum, and then divides the sum by world size. When computing sum, it may cause overflow.

Components that may involve changes:

Brief description of your proposal if any: Currently we do not have a perfect solution for this.