MOSS icon indicating copy to clipboard operation
MOSS copied to clipboard
verified publisher icon OpenMOSS

How to train a custom tokenizer for Chinese from scratch

Open SparkJiao opened this issue 2 years ago • 0 comments

Hi, wonderful work!

May I know how to train a custom tokenizer for Chinese from scratch? Is there any public reference or code can share?

Thanks for your help very much!

best, Fangkai

SparkJiao avatar Jul 07 '23 07:07 SparkJiao