openspeech icon indicating copy to clipboard operation
openspeech copied to clipboard

Add huggingface tokenizer

Open upskyy opened this issue 3 years ago • 0 comments

🚀 Add huggingface tokenizer

image

tokenizers : https://github.com/huggingface/tokenizers

Motivation

  • We can tokenize easily and quickly using tokenizers.
  • We can easily apply various tokenizers.

Your contribution

I will implement.

upskyy avatar Jan 04 '22 07:01 upskyy