lambda-networks icon indicating copy to clipboard operation
lambda-networks copied to clipboard

Published 20 hours ago • verified publisher icon lucidrains

Positional Encoding

Open JeremyCJM opened this issue 4 years ago • 2 comments

Why the positional encoding is random?

JeremyCJM avatar Feb 17 '21 18:02 JeremyCJM

maybe ask @khanrc https://github.com/lucidrains/lambda-networks/commit/1ac51c0ce52de0ba78e2ab97152d2bafe2543b95#diff-6d46aff567baa42dfd4c6b8d2f5c956d3ea84b2811ca663ea67159e323344ccb

johndpope avatar Feb 23 '21 20:02 johndpope

It is learnable positional embedding, which is initialized randomly and learned during training. Please refer to the paper.

khanrc avatar Feb 24 '21 05:02 khanrc