keras-nlp icon indicating copy to clipboard operation
keras-nlp copied to clipboard

Published 20 hours ago • verified publisher icon keras-team

Integrate ELECTRA Pretrained Model into Keras_NLP Ecosystem

Open pranavvp16 opened this issue 1 year ago • 6 comments

I would like to add ELECTRA pretrained model in the keras_nlp ecosystem. I have went through the CONTRIBUTINT_MODELS.md and making an issue is the first task to add the model. I have pretty good understanding of transformers and have went through the ELECTRA architecture mentioned here. So if there is no problem regarding the integration I can start working on this !!

pranavvp16 avatar Oct 24 '23 13:10 pranavvp16

Electra would be a welcome addition! Thanks! As mentioned in the model contribution guide, first thing to do would be to write a backbone and a checkpoint conversion script from a known source that shows we can match outputs.

Huggingface might be the easiest checkpoint source. The original electra repo would be totally fine as well, but looks like the implementation is in tf1, which would be a little painful to work with.

mattdangerw avatar Oct 25 '23 20:10 mattdangerw

Thanks for assigning the issue, I'll start working on it

pranavvp16 avatar Oct 27 '23 15:10 pranavvp16

@mattdangerw I had one Colab floating around which would implement electra in KerasNLP, will share if I find . Would be helpful for @pranavvp16

shivance avatar Oct 30 '23 04:10 shivance

#794

shivance avatar Oct 30 '23 11:10 shivance

@shivance thanks for the notebook but I think the notebook shares the pre-training approach of ELECTRA while I'm trying to implement the backbone of ELECTRA which is similar to BERT with some changes. Please let me know if I'm wrong here cause I'm pretty new to this thing. I have implemented the backbone successfully in keras according to the ELECTRA architecture, and working of weight conversion from hugging face checkpoint

pranavvp16 avatar Oct 30 '23 14:10 pranavvp16

I'm trying to implement the backbone of ELECTRA which is similar to BERT with some changes. Please let me know if I'm wrong here cause I'm pretty new to this thing. I have implemented the backbone successfully in keras according to the ELECTRA architecture, and working of weight conversion from hugging face checkpoint

Yes that's definitely a good approach! Let's start with the backbone, not the pretraining approach.

The pretraining approach would make for an excellent keras.io example (which was https://github.com/keras-team/keras-nlp/issues/794), but beginning with a backbone is the right first step on adding this model to our API.

Will take a look at the issues on the PR shortly!

mattdangerw avatar Oct 31 '23 06:10 mattdangerw