CPT pretrain problem

Open SunyanGu opened this issue 3 years ago • 3 comments

when I use roberta_zh to pretrain CPT model , raise error "Error(s) in loading state_dict for BertModel". So what pretrained model shoud i use? Roberta or BERT ?

Mar 15 '23 07:03 SunyanGu

If you use roberta_zh with transformers, it would have the same architecture as BERT. And it should work.

Mar 15 '23 09:03 choosewhatulike

I'm sure that I use transformers to load roberta_zh.But the model which I download have error parameter name. Can you give me a link to download right roberta_zh?

Mar 16 '23 11:03 SunyanGu

This should work: https://huggingface.co/hfl/chinese-roberta-wwm-ext

Mar 16 '23 13:03 choosewhatulike