GLM icon indicating copy to clipboard operation
GLM copied to clipboard

Add special_token

Open chenzebiaohub opened this issue 10 months ago • 0 comments

# special_tokens_dict = {'additional_special_tokens': ['<ENTITY>', '<DESCRIPTION>']}
# tokenizer.add_special_tokens(special_tokens_dict)
# 模型加载
# model = AutoModelForSeq2SeqLM.from_pretrained(conf.model_folder, trust_remote_code=True)
**# model.resize_token_embeddings(len(tokenizer))  # 关键步骤,重新调整所有的向量数量,即vocab size**
# tokenizer.save_pretrained(conf.output_path)

报错resize_token_embeddings [PyTroch报错NotImplementedError]

chenzebiaohub avatar Apr 19 '24 06:04 chenzebiaohub