keras-team
I want to provide the Keras_hub implementation of RoFormer.
https://github.com/ZhuiyiTechnology/roformer Roformer is a BERT-like model. It adds the now very commonly used Rope position encoding on top of BERT. In fact, this is the first practical application of Rope position encoding. I found that Keras_hub lacks a powerful Chinese BERT-like model. And RoFormer happens to be a native Chinese BERT model, and its architecture is very similar to that of Modern BERT. This will also be helpful for future implementations related to Modern BERT.
Currently, there are two models, Bert and XLMroberta, which have Chinese and multilingual versions. However, one problem is that they have a limit on the length, making it difficult to meet the needs of many modern long-text applications.
https://github.com/ZhuiyiTechnology/roformer-v2 (Sorry, the webpage only has a Chinese interface.) I found that RoFormer also has a more powerful v2 version, which doesn't have a corresponding implementation in HF, but its performance is better.
I tend to think we can directly implement this version, which offers a large base and a small version. Compared to previous versions, it may be more applicable.
Hi @pass-lin, Thank you for your contribution to Keras-Hub. Can we mark this issue as completed?
