DeepLearningExamples icon indicating copy to clipboard operation
DeepLearningExamples copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

Electra-small pretraining

Open 73minerva opened this issue 2 years ago • 1 comments

Hey,

I want to pretrain and benchmark small and base versions of Electra for the Arabic and Persian languages. As mentioned in the run_pretraining python file, only "base" and "large" model_size are supported. Although it seems that in fine-tuning, the Electra-small model is supported. Is the implementation of pretraining for Electra-small in your future plans? If it's not, what about a PR ?!

73minerva avatar Mar 04 '22 09:03 73minerva

The small model might work if you can come up with a config for it based on the model parameters.

A PR would definitely be appreciated. Feel free to make one.

sharathts avatar Mar 04 '22 10:03 sharathts