Pretrained-Language-Model icon indicating copy to clipboard operation
Pretrained-Language-Model copied to clipboard
verified publisher icon huawei-noah

TinyBERT for masked LM

Open danchern97 opened this issue 3 years ago • 0 comments

Hi! I've been trying to measure MLM perplexity for TinyBERT model (in particular, tinybert6l), and I keep getting inconsistent results. Looks like the MLM head for TinyBERT is not loaded properly when loading from AutoModelForMaskedLM or by BertForMaskedLM.

danchern97 avatar Jul 18 '22 10:07 danchern97