The mathematical trick in self-attention, why it returns false for torch.allclose(xbow, xbow2)?

[Ryan-ZL-Lin]

Hi I noticed that the result of torch.allclose(xbow, xbow2), torch.allclose(xbow, xbow3) are all false when running the Collab example gpt-dev.ipynb in The mathematical trick in self-attention section. Here is what I got, has anyone encountered the same issue?

[0xArwa]

@Ryan-ZL-Lin You can adjust the relative tolerance for less strict comparison. the default value is 1e-05 in PyTorch 2.2

This snippet will output True

torch.allclose(xbow, xbow2, rtol= 1e-04) # default 1e-05

[yyinsomnia]

我也详细排查了一下，发现太小的值，这里[1,5,1] 是 0.0020，会导致allclose判断为False 这个问题很有意思，我记得21年跑这个代码没有这个问题，现在出现了。我也运行了一下Andrej的原始notebook也是Fasle 说明大概率是python和torch的版本升级导致的？