keras-preprocessing icon indicating copy to clipboard operation
keras-preprocessing copied to clipboard

how keras.preprocessing.text.Tokenizer processing oov_token and predefined special token?

Open kingweiliu opened this issue 7 years ago • 0 comments

I try to use Tokenizer to handle string input. "oov_token" param is given "<UNK>" when Tokenizer was initializing. However, oov_token's corresponding index is more than num_words. This index can't be used directly in embedding_lookup by token index. Another question is how to use predefined words with Tokenizer , such as <GO> <EOS> <PAD> .

kingweiliu avatar Nov 23 '18 14:11 kingweiliu