The accuracy is misaligned when using bf16

[slowlyC]

trafficstars

🐛 Describe the bug

It is normal when using fp16, but there are some problems with bf16. It seems that because swiglu uses tl.sigmoid(), it needs to be converted to fp32, but there is a loss of accuracy when converting to bf16.

This part of the code: @triton.jit def silu(x): return x * tl.sigmoid(x)

Reproduce

No response

Versions

v0.4.2

[Irvingwangjr]

I think I also met this problem