BFloat16 support in multi_tensor_*

[zhengwy888]

Hi,

I was trying to use Apex (under fairseq) with BFloat16 and this error pop up.

 multi_tensor_total_norm
    norm = multi_tensor_l2norm(
RuntimeError: "multi_tensor_l2norm_cuda" not implemented for 'BFloat16'

I believe it's because this dispatch kernel doesn't include BFloat16. https://github.com/NVIDIA/apex/blob/master/csrc/multi_tensor_l2norm_kernel.cu#L326

Is there any plan to include bf16 support? Thank you.

[yuvalkirstain]

I also encountered this issue and am interested in the question :)

[yiakwy-xpu-ml-framework-team]

@zhengwy888 @yuvalkirstain bf has already been supported in apex aten::Tensor api https://github.com/NVIDIA/apex/pull/1407/files

Hence I believe this issue should be closed ?