Baichuan-13B icon indicating copy to clipboard operation
Baichuan-13B copied to clipboard

Published 20 hours ago • verified publisher icon baichuan-inc

tokenizer.tokenize("<reserved_102>")

Open huangqingyi-code opened this issue 1 year ago • 0 comments

你好，tokenizer.tokenize("<reserved_102>")时为什么会多出来一个'▁'符号。tokenizer.encode导致时多了一个31106id

huangqingyi-code avatar Jul 25 '23 03:07 huangqingyi-code