lingo-xp

Does it support chinese tokenizer or custom tokenizer?

2

**Describe the solution you'd like** Is there any plan to support chinese tokenizer? I just check the document , but find it doesnt support it now. Maybe we can supprt...