LLaVA icon indicating copy to clipboard operation
LLaVA copied to clipboard

Published 20 hours ago • verified publisher icon haotian-liu

[Question] layer_norm float32?

Open ByungKwanLee opened this issue 1 year ago • 0 comments

Question

Is there any reason for 'norm' to set float32?

image

what if it is set to bfloat16?

Do you think the difference between float32 and bfloat16 in 'norm' operator makes performance gap be bigger?

ByungKwanLee avatar Jan 21 '24 05:01 ByungKwanLee