Where are the correspondent codes to the alpha, beta, miu and sigma?

[fishfree]

As you can see here.

[floriankark]

Those are parameters of Layernorm.

https://github.com/karpathy/nanoGPT/blob/eba36e84649f3c6d840a93092cb779a260544d08/model.py#L18C1-L27C84

For a better understanding see: https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html and more indepth https://arxiv.org/abs/1607.06450