GuenKainto

Hello, can I ask how to create a tokenizer file for Japanese? I see that Japanese people use some Kanji in sentences or words. I found a simple tokenizer file...

> @aklacar1 you might need to modify a few things to support a larger tokenizer, but keeping the tokenizer at 256 will work out of the box , if you...

Thanks for replying

yeb, but with this we can't move grid any more, it locked :")