ZRKGC icon indicating copy to clipboard operation
ZRKGC copied to clipboard

special tokens of "<#Q2K#>"

Open jind11 opened this issue 4 years ago • 0 comments

Hi, I tried to run this code but encountered one problem: there are several special tokens such as "<#Q2K#>", "<#K#>", and "<#Q#>" after bert tokenization, however, the original bert vocab does not contain these three tokens, which caused the tokenization indexing error. How did you solve this issue? Thanks!

jind11 avatar Oct 09 '20 06:10 jind11