Bert-Chinese-Text-Classification-Pytorch icon indicating copy to clipboard operation
Bert-Chinese-Text-Classification-Pytorch copied to clipboard

Published 20 hours ago • verified publisher icon 649453932

Question about masked language model

Open paprika0741 opened this issue 2 years ago • 0 comments

代码中masked language modeling labels中-1标记的是被masked的token，loss计算忽略被mask的token，但是BERT论文中写的是”the final hidden vectors corresponding to the mask tokens are fed into an output softmax over the vocabulary“ 只计算masked token处的loss

paprika0741 avatar Sep 23 '23 13:09 paprika0741