BERT-pytorch icon indicating copy to clipboard operation
BERT-pytorch copied to clipboard

embedding/position.py with RuntimeError if d_model is odd

Open bookong22 opened this issue 4 years ago • 2 comments

such as if set d_model = 29, max_len = 10000 RuntimeError: The expanded size of the tensor (14) must match the existing size (15) at non-singleton dimension 1. Target sizes: [10000, 14]. Tensor sizes: [10000, 15] code in this line may be fixed (line 30): pe[:, 1::2] = torch.cos(position * div_term)

bookong22 avatar Jun 15 '20 03:06 bookong22

modify pe[:, 1::2] = torch.cos(position * div_term) to pe[:, 1::2] = torch.cos(position * div_term)[:, :d_model//2]

bookong22 avatar Jun 15 '20 03:06 bookong22

d_model should be a even number

cheng940929 avatar Jun 30 '22 06:06 cheng940929