transformers Ernie-M

Model description

Ernie-M looks pretty good in multilingual benchmarks, beating XLM-Roberta. Paddlepaddle recently added ernie-m to the huggingface repo, we can use it with paddlenlp.transformers. Would be nice to have the model supported in huggingface transformer as well.

Open source status

[X] The model implementation is available
[X] The model weights are available

Provide useful links for the implementation

https://huggingface.co/PaddlePaddle/ernie-m-base https://huggingface.co/PaddlePaddle/ernie-m-large

Jan 14 '23 10:01 KnutJaegersberg

https://github.com/PaddlePaddle/ERNIE/blob/ernie-kit-open-v1.0/erniekit/modules/ernie.py

has more implementation details.

Jan 17 '23 17:01 shermansiu

Hi, @shermansiu is there any pytorch/tf implementation of this model?

Jan 20 '23 18:01 susnato

None that I'm aware of.

Anyways, the author of ERNIE-Pytorch ported over a few other Ernie models to Huggingface. I'm sure it could be adapted for this. And the PaddlePaddle syntax is quite similar to that of PyTorch, so I'm sure it should be relatively easy, though it'll probably take some time.

Jan 20 '23 18:01 shermansiu

@shermansiu Thanks for the resources! I am currently trying to port the model to huggingface(pytorch), (done till Embedding Layer with acceptable tolerance of 1e-3)

Jan 23 '23 18:01 susnato

Hi @KnutJaegersberg, Ernie-M is implemented!

Feb 16 '23 07:02 susnato