BertTokenizers icon indicating copy to clipboard operation
BertTokenizers copied to clipboard

Published 20 hours ago • verified publisher icon NMZivkovic

The tokenization for Korean text seems not correct.

Open terryqj0107 opened this issue 1 year ago • 0 comments

Is the tokenizer implementation the same as python version? It seems that it can not cut the Korean text into correct wordpiece.

terryqj0107 avatar May 12 '23 02:05 terryqj0107