minimind icon indicating copy to clipboard operation
minimind copied to clipboard

Published 20 hours ago • verified publisher icon jingyaogong

关于自训tokenizer的问题

Open Harris-Xie opened this issue 1 month ago • 0 comments

想问下tokenizer_train的jsonl文件里为什么不用中文呢，这样不会导致中文编码效率过低的问题嘛？

Harris-Xie avatar Jan 08 '25 19:01 Harris-Xie