Anthony MOI
Anthony MOI
Alright, that's what I feared. This is happening because you have a single tokenizer, that is used by 2 different threads. While the tokenizer is encoding (on one thread), if...
Indeed, at the moment `tokenizers` does not support SentencePiece models that use the Unigram model, but only the BPE model. The support for Unigram is the next thing on the...
Hi @aqibsaeed, please read my answer in another issue that should give you some direction on how to do this. Here is the answer in question: https://github.com/huggingface/tokenizers/issues/243#issuecomment-617860020 There is a...