A question about the code of " BertTrainDataset.__getitem__(self, index)" function

[JiekeLi]

I want to ask a question that why you give '0' label to the none masked token when generate a example here:

    for s in seq:
           prob = self.rng.random()
           if prob < self.mask_prob:
               prob /= self.mask_prob

               if prob < 0.8:
                   tokens.append(self.mask_token)
               elif prob < 0.9: 
                   tokens.append(self.rng.randint(1, self.num_items))
               else: 
                   tokens.append(s)

               labels.append(s) 
           else:
               tokens.append(s)
               labels.append(0) # **why give 0 label here and not the index of s ?**

Thanks !

[thomalm]

I was also wondering the same. Masking should only be applied to input tokens

[reckzhou]

because loss function is CrossEntropyLoss(ignore_index=0) ，it ignored index for zero，only compute loss for mask item or random item。

but :

if prob < 0.8: 
    tokens.append(self.mask_token)
    labels.append(s)
elif prob < 0.9: 
    tokens.append(self.rng.randint(1, self.num_items))
    labels.append(s)
else: 
    tokens.append(s)
    labels.append(0)  #? I changed it.

[kyeongchan92]

And why divide one more time using 0.8, 0.9 after mask_prob? Random item insertion on 0.15 * 0.1 ? I couldn't find this part on paper