SHI-Labs
RoPE vs RPB
Hello,
First and foremost, thank you for your excellent work on this project.
I've noticed that support for Relative Position Bias (RPB) was removed starting with version 0.20.0. I would like to kindly request if you would consider re-enabling it.
The reason for this request is that models using RPB demonstrate significantly better performance when trained on a single resolution and then used for inference across a wide variety of different resolutions. With RPB, the relative positions within the attention window are always consistent, meaning these relationships are always well-trained.
In contrast, when changing resolutions at inference time, a model trained with RoPE tends to show a degradation in performance if it hasn't been exposed to those resolutions during training.
Thank you for your time and consideration.
