ALBERT-Pytorch
ALBERT-Pytorch copied to clipboard
graykode
Pytorch Implementation of ALBERT(A Lite BERT for Self-supervised Learning of Language Representations)
if i wanna use my own textual data to pre-train a electra from scatch, what is the format of the text? Only sentence segmentation or even more ?? Please help.
hey there, sadly i dont clearly understand how the wiki.train.tokens and vocab.txt file is produced. followingly could you please show me the steps on how to reproduce the same train.tokens...
what should be my mask_alpha and mask_beta values if my seqence lenght is about 10-20?
i am trying to train albert model using my own dataset i don't know why got the error like this please let me know, when you figure out thank you
I'm running classify on the MRPC dataset. In trainer.train trainer.train(get_loss,model_file,True), it allows only three arguments not 4 so I cant use the pretrain file. Also it runs out of memory,...
In ALBERT(Lan at el), There is not detail about 80% mask  But, from n-gram masking (Joshi et al., 2019), they said about 80/10/10 > As in BERT, we also...
