openspeech
openspeech copied to clipboard
Add huggingface tokenizer
🚀 Add huggingface tokenizer

tokenizers : https://github.com/huggingface/tokenizers
Motivation
- We can tokenize easily and quickly using tokenizers.
- We can easily apply various tokenizers.
Your contribution
I will implement.