ERNIE-Pytorch icon indicating copy to clipboard operation
ERNIE-Pytorch copied to clipboard
verified publisher icon nghuyong

Tokenizer中间是否缺少一个token？

Open LiShaoyu5 opened this issue 2 years ago • 0 comments

训练过程中提示： The OrderedVocab you are attempting to save contains a hole for index 12084, your vocabulary could be corrupted ! 检查了tokenizer.json和vocab.txt，’: 12083,{: 12085之间确实没有12084，请问这是tokenizer本身如此吗？

LiShaoyu5 avatar Jul 18 '23 08:07 LiShaoyu5