keras-nlp icon indicating copy to clipboard operation
keras-nlp copied to clipboard

Published 20 hours ago • verified publisher icon keras-team

Fix BytePair special tokens tokenization

Open abuelnasr0 opened this issue 1 year ago • 2 comments

BytePair already tokenize special tokens but it was having a small nit explained here keras-team/keras-nlp#1435 this PR fixes it.

abuelnasr0 avatar Feb 20 '24 13:02 abuelnasr0