spert
spert copied to clipboard
Does SpERT work with GPT models?
Hi, just wondering whether there are any efforts in the SpERT + GPT model integration?
Hi,
I think nothing speaks against simply replacing BERT with GPT. Just have a look at the huggingface library (which we are using for the BERT implementation) for best practices. You especially need to replace the BERT related calls in spert/models.py
and BertTokenizer/BertConfig in spert/spert_trainer.py
. Other than that, the preprocessing, training and model architecture related code should also work with GPT.