ConvBert-PyTorch

🤗An unofficial PyTorch implementation of ConvBert based on huggingface/transformers.

🤗基于huggingface/transformers的ConvBert的非官方PyTorch实现。

requirements

1. transformers==3.5.1
2. torch>=1.5.0
3. wandb

pretrain

python run_language_modeling.py

Reference

1. ConvBERT: Improving BERT with Span-based Dynamic Convolution
2. yitu-opensource/ConvBert
3. huggingface/transformers
4. JunnYu/ConvBert_huggingface
5. sjpark9503/ELECTRA-pretraining
6. ThilinaRajapakse/simpletransformers