Arthur

Hey! Would recommend you to ask on the forum as you are using custom code. However note that printing / debugging to get what you pass the tokenizer is usually...

Hey! That is not really the way to make it ! Are you still interested in having the fast version?

Sorry but I have not idea how to go back to sentencepiece 😅 the format is not super open. What is the motivation?

Hey! WOuld you like to open a PR for a fix?

Maybe using this one: https://huggingface.co/datasets/Salesforce/wikitext or one that is on the hub would be nice!

#30868 is the new PR

No Idea @Xe ! And yes I know, I will look into the fix this week! Sorry all 🤗

Packing is planned

Most probably not next release, but the one after that!

#31446 for packing