Junseong Kim comments

Results 46 comments of


                                            Junseong Kim

Making Book Corpus

@mapingshuo Sorry It's my fault. haha I just made that title in 5seconds :) thank you!! 👍

when training the masked LM, the unmasked words (have label 0) were trained together with masked words?

@coddinglxf I just solved that problem with `nn.NLLLoss(ignore_index=0)` which 0 is equal to pad_index. Even if we target the 0(unmasked_value), it doesn't affect to the loss of propagation

when training the masked LM, the unmasked words (have label 0) were trained together with masked words?

@coddinglxf that's what I thought at first, but can't implement it efficiently as much as GPU computation time. If you have any idea please implement and pull request plez :)...

when training the masked LM, the unmasked words (have label 0) were trained together with masked words?

@leon-cas yes #36 it's solved with your question

Vocab Replace \t to blank issue

I'll update the vocab builder ASAP! thanx

Making Wikipedia Corpus

#32

Tie the input and output embedding?

hmmm? what do you mean the output embedding? you mean the softmaxed output distribution?

Tie the input and output embedding?

Is there any benefit if we bind two layer weight? If it is, please can you let me know some references which has similar architecture?

Tie the input and output embedding?

@jiqiujia @briandw Cool I'll implement is on 0.0.1a5 version, but it seems like solving #32 is more high priority

what’s your data set?

@iOSGeekerOfChina I didn't decide yet, just started this project one hour ago haha. Do you think using the dataset which referred on paper is good idea? Or have some another...