bytepiece icon indicating copy to clipboard operation
bytepiece copied to clipboard
verified publisher icon bojone

弱问下训练一个tokenizer的训练目标是什么？

Open guotong1988 opened this issue 2 years ago • 1 comments

多谢！

guotong1988 avatar Sep 10 '23 09:09 guotong1988

同等vocab_size时压缩率最高。

bojone avatar Sep 10 '23 12:09 bojone