How to understand the positional embedding part in rvt(rotary embedding)?

[CiaoHe]

Hi, I'm just curious about the different implement way between the AxialRotaryEmbedding and the original ones.

Since I notice u multiply the pi when calculating sin and cos vector. But in the original paper, they don't use the pi. And I got confused when computing the sin and cos vector. Could u explain a little further?

Thanks a lot. I really learn a lot from ur work

[lucidrains]

@CiaoHe hello! yea sure, so the frequencies in the original paper were designed for language modeling, but I ended up using the frequencies as defined in the Perceiver paper https://arxiv.org/abs/2103.03206 I figured if they made it work well in that setting, it should work well here. Have you found RVT to work well?

[CiaoHe]

@CiaoHe hello! yea sure, so the frequencies in the original paper were designed for language modeling, but I ended up using the frequencies as defined in the Perceiver paper https://arxiv.org/abs/2103.03206 I figured if they made it work well in that setting, it should work well here. Have you found RVT to work well?

Hi Phil, thanks for replying. I'm still trying to understand all components of the Alphafold2 you implemented, so I just read codes related to that (include this rotaryEmbedding). Later I will check whether this RVT work well. But anyway, thanks a. lot, ur codes teach me a lot since I just a new to the this area.