ChatGLM-Efficient-Tuning
ChatGLM-Efficient-Tuning copied to clipboard
Published
20 hours ago •
hiyouga
hiyouga
ChatGLM-6b lora微调出错，label ids全为-100
使用的命令如下，并且更换了数据集也是一样的情况。
CUDA_VISIBLE_DEVICES=0 python src/train_sft.py \
--do_train \
--dataset alpaca_gpt4_zh \
--dataset_dir data \
--finetuning_type lora \
--output_dir finetunev1_alpaca \
--overwrite_cache \
--per_device_train_batch_size 4 \
--gradient_accumulation_steps 4 \
--lr_scheduler_type cosine \
--logging_steps 10 \
--save_steps 1000 \
--learning_rate 5e-5 \
--num_train_epochs 3.0 \
--plot_loss \
--fp16 \
--model_name_or_path /root/autodl-tmp/models/chatglm-6b
你使用的是 ChatGLM 还是 ChatGLM2？如果是2必须要加 --use_v2 参数
是chatglm，出现-100的问题。
v2的话，使用羊驼数据集lora微调，参数和上面相似（加use_v2），会有一定概率复读。
GLM2最新代码，freeze训练保存时，会出现以下报错：
06/28/2023 15:30:10 - INFO - utils.peft_trainer - Saving model checkpoint to ./output/cognition
[INFO|configuration_utils.py:458] 2023-06-28 15:30:12,155 >> Configuration saved in ./output/cognition/config.json
[INFO|configuration_utils.py:364] 2023-06-28 15:30:12,155 >> Configuration saved in ./output/cognition/generation_config.json
[INFO|modeling_utils.py:1829] 2023-06-28 15:30:14,060 >> Model weights saved in ./output/cognition/pytorch_model.bin
[INFO|tokenization_utils_base.py:2181] 2023-06-28 15:30:14,275 >> tokenizer config file saved in ./output/cognition/tokenizer_config.json
[INFO|tokenization_utils_base.py:2188] 2023-06-28 15:30:14,275 >> Special tokens file saved in ./output/cognition/special_tokens_map.json
Traceback (most recent call last):
File "src/train_sft.py", line 105, in <module>
main()
File "src/train_sft.py", line 77, in main
trainer.save_model()
File "/root/miniconda3/envs/llm/lib/python3.8/site-packages/transformers/trainer.py", line 2866, in save_model
self._save(output_dir)
File "/root/autodl-tmp/workspace-liwh/ChatGLM-Efficient-Tuning/src/utils/peft_trainer.py", line 114, in _save
self.tokenizer.save_pretrained(output_dir)
File "/root/miniconda3/envs/llm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2192, in save_pretrained
save_files = self._save_pretrained(
File "/root/miniconda3/envs/llm/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2240, in _save_pretrained
vocab_files = self.save_vocabulary(save_directory, filename_prefix=filename_prefix)
File "/root/.cache/huggingface/modules/transformers_modules/chatglm2-6b/tokenization_chatglm.py", line 137, in save_vocabulary
with open(self.vocab_file, 'rb') as fin:
AttributeError: 'ChatGLMTokenizer' object has no attribute 'vocab_file'