DeepLearningExamples
DeepLearningExamples copied to clipboard
Electra-small pretraining
Hey,
I want to pretrain and benchmark small and base versions of Electra for the Arabic and Persian languages. As mentioned in the run_pretraining python file, only "base" and "large" model_size are supported. Although it seems that in fine-tuning, the Electra-small model is supported. Is the implementation of pretraining for Electra-small in your future plans? If it's not, what about a PR ?!
The small model might work if you can come up with a config for it based on the model parameters.
A PR would definitely be appreciated. Feel free to make one.