GLM
GLM copied to clipboard
Add special_token
# special_tokens_dict = {'additional_special_tokens': ['<ENTITY>', '<DESCRIPTION>']}
# tokenizer.add_special_tokens(special_tokens_dict)
# 模型加载
# model = AutoModelForSeq2SeqLM.from_pretrained(conf.model_folder, trust_remote_code=True)
**# model.resize_token_embeddings(len(tokenizer)) # 关键步骤,重新调整所有的向量数量,即vocab size**
# tokenizer.save_pretrained(conf.output_path)
报错resize_token_embeddings [PyTroch报错NotImplementedError]