how-to-train-tokenizer icon indicating copy to clipboard operation
how-to-train-tokenizer copied to clipboard

Published 20 hours ago • verified publisher icon yanqiangmiffy

falcon词表扩充

Open belle9217 opened this issue 11 months ago • 0 comments

你好，我用该项目训练中sentencepiece训练了一个中文词表，和falcon的英文词表无法合并，使用AutoTokenizer加载的falcon英文词表，没有sp_model属性，请问该怎么解决呢

belle9217 avatar Aug 04 '23 06:08 belle9217