markov-discord icon indicating copy to clipboard operation
markov-discord copied to clipboard

Published 20 hours ago • verified publisher icon claabs

Unable to learn Japanese

Open AI-Avalon opened this issue 2 years ago • 1 comments

I cannot learn Japanese. If you don't mind, could you support Japanese as well?

AI-Avalon avatar Apr 26 '22 13:04 AI-Avalon

The current setup uses space (" ") to separate words for training. I found this paper about Japanese word segmentation: https://aclanthology.org/H94-1054.pdf

It seems much more complicated than the space delimited approach. Since I don't know Japanese, there's not much I can help with.

claabs avatar Apr 26 '22 14:04 claabs