keras-nlp icon indicating copy to clipboard operation
keras-nlp copied to clipboard

Published 20 hours ago • verified publisher icon keras-team

Support tokenization of special tokens for sentencepiece tokenizer

Open abuelnasr0 opened this issue 1 year ago • 2 comments

This PR enables tokenization of special tokens for SentencePieceTokenizer as was suggested in keras-team/keras-nlp#1395 and it's a follow up of this PR keras-team/keras-nlp#1397 .

abuelnasr0 avatar Feb 19 '24 20:02 abuelnasr0