pytorch-original-transformer
pytorch-original-transformer copied to clipboard
Frequency in the positional encodings
What does the frequency represent in positional encoding ? Why do we need to multiply it with the positional values?
frequencies = torch.pow(10000., -torch.arange(0, model_dimension, 2, dtype=torch.float) / model_dimension)