Cagri Toraman

vocab_size issue with Whitespace pre_tokenizer

9

Hi, I train a WordPiece tokenizer with a custom vocabulary size. But somehow the vocab size of my trained tokenizer gets much higher than my input size ( ```16700```). ```...