ERNIE-Pytorch
ERNIE-Pytorch copied to clipboard
nghuyong
ERNIE Pytorch Version
ERNIE-Pytorch
This project is to convert ERNIE series models from paddlepaddle to huggingface's format (in Pytorch).
Get Started
Take ernie-1.0-base-zh as an example:
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh")
model = BertModel.from_pretrained("nghuyong/ernie-1.0-base-zh")
Supported Models
| Model Name | Language | Description |
|---|---|---|
| ernie-1.0-base-zh | Chinese | Layer:12, Heads:12, Hidden:768 |
| ernie-2.0-base-en | English | Layer:12, Heads:12, Hidden:768 |
| ernie-2.0-large-en | English | Layer:24, Heads:16, Hidden:1024 |
| ernie-3.0-base-zh | Chinese | Layer:12, Heads:12, Hidden:768 |
| ernie-3.0-medium-zh | Chinese | Layer:6, Heads:12, Hidden:768 |
| ernie-3.0-mini-zh | Chinese | Layer:6, Heads:12, Hidden:384 |
| ernie-3.0-micro-zh | Chinese | Layer:4, Heads:12, Hidden:384 |
| ernie-3.0-nano-zh | Chinese | Layer:4, Heads:12, Hidden:312 |
| ernie-health-zh | Chinese | Layer:12, Heads:12, Hidden:768 |
| ernie-gram-zh | Chinese | Layer:12, Heads:12, Hidden:768 |
You can find all the supported models from huggingface's model hub: huggingface.co/nghuyong, and model details from paddle's official repo: PaddleNLP and ERNIE.
Note for ERNIE-3.0
If you want to use ernie-3.0 series models, you need to add task_type_id to BERT model following this MR
OR you can re-install the transformers from my changed branch.
pip uninstall transformers # optional
pip install git+https://github.com/nghuyong/transformers@add_task_type_id # reinstall, 4.22.0.dev0
Then you can load ERNIE-3.0 model as before:
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained("nghuyong/ernie-3.0-base-zh")
model = BertModel.from_pretrained("nghuyong/ernie-3.0-base-zh")
Details
I want to convert the model from paddle version by myself 😉
The following will take ernie-1.0-base-zh as an example to show how to convert.
- Download the paddle-paddle version ERNIE model from here , move to this project path and unzip the file.
-
pip install -r requirements.txt -
python convert.py - Now, a folder named
convertwill be in the project path, and there will be three files in this folder:config.json,pytorch_model.binandvocab.txt.
I want to check the calculation results before and after model conversion 😁
python test.py --task logit_check
You will get the output:
huggingface result
pool output: [-1. -1. 0.9981035 -0.9996652 -0.78173476 -1. -0.9994901 0.97012603 0.85954666 0.9854131 ]
paddle result
pool output: [-0.99999976 -0.99999976 0.9981028 -0.9996651 -0.7815545 -0.99999976 -0.9994898 0.97014064 0.8594844 0.985419 ]
It can be seen that the result of our convert version is the same with the official paddlepaddle's version.
I want to reproduce the cloze test in ERNIE1.0's paper 😆
python test.py --task cloze_check
You will get the output:
huggingface result
prediction shape: torch.Size([47, 18000])
predict result: ['西', '游', '记', '是', '中', '国', '神', '魔', '小', '说', '的', '经', '典', '之', '作', '，', '与', '《', '三', '国', '演', '义', '》', '《', '水', '浒', '传', '》', '《', '红', '楼', '梦', '》', '并', '称', '为', '中', '国', '古', '典', '四', '大', '名', '著', '。']
[CLS] logit: [-15.693626 -19.522263 -10.429456 ... -11.800728 -12.253127 -14.375117]
paddle result
prediction shape: [47, 18000]
predict result: ['西', '游', '记', '是', '中', '国', '神', '魔', '小', '说', '的', '经', '典', '之', '作', '，', '与', '《', '三', '国', '演', '义', '》', '《', '水', '浒', '传', '》', '《', '红', '楼', '梦', '》', '并', '称', '为', '中', '国', '古', '典', '四', '大', '名', '著', '。']
[CLS] logit: [-15.693538 -19.521954 -10.429307 ... -11.800765 -12.253114 -14.375412]
Citation
If you use this work in a scientific publication, I would appreciate that you can also cite the following BibTex entry:
@misc{nghuyong2019@ERNIE-Pytorch,
title={ERNIEPytorch},
author={Yong Hu},
howpublished={\url{https://github.com/nghuyong/ERNIE-Pytorch}},
year={2019}
}
nghuyong