pytorch-transformer icon indicating copy to clipboard operation
pytorch-transformer copied to clipboard

Fix Layernorm Implementation

Open Xue10 opened this issue 1 year ago • 0 comments

According to the formula norm = (x - mean) / sqrt(var + eps) not (x - mean)/(std + eps) sqrt(var + eps) == sqrt(std**2 + eps) != (std + eps) Though the difference may be small, it is not a strictly correct implementation of LayerNorm.

Xue10 avatar Sep 30 '24 01:09 Xue10