1-billion-word-language-modeling-benchmark icon indicating copy to clipboard operation
1-billion-word-language-modeling-benchmark copied to clipboard

Published 20 hours ago • verified publisher icon ciprian-chelba

if the word not in vocab， what should I do？ or it always can't happen because the FullTokenizer

Open wangwang110 opened this issue 6 years ago • 1 comments

wangwang110 avatar Dec 14 '18 02:12 wangwang110

Please read https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark/blob/master/README.perplexity_and_such and let me know if you still have questions.

ciprian-chelba avatar Dec 14 '18 17:12 ciprian-chelba