AttributeError: 'Namespace' object has no attribute 'emb_size'

[zhilonglu]

Firstly i download the data and run the preprocess.py, it generate the .pt, when I try to run the pretrain.py to finetune on my own insrtuction data, the error occurs, how to solve it ,thanks!

[fengyh3]

can you show more details about your pretraining? for example running bash. It seems your model config has some problems.

[zhilonglu]

python pretrain.py --pretrained_model_path /home/pretrain_models/LLaMA-zh-7b-hf --dataset_path dataset.pt --spm_model_path /home/pretrain_models/LLaMA-zh-7b-hf/tokenizer.model --config_path /home/pretrain_models/LLaMA-zh-7b-hf/config.json --output_model_path output/llama_zh_7b --world_size 1 --data_processor alpaca --total_steps 20000 --save_checkpoint_steps 100 --batch_size 24 This is the command I want to pretrain the model, dataset.pt was generated by preprocess.py. Actually , i change the json Line 4 in tencentpretrain/utils/constants.py to llama_special_tokens_map.json

[fengyh3]

the version of xxx-hf is for huggingface transformers, so if you want to use TencentPretrain to do this job, you should choose the tencentpretrain version, for example, https://huggingface.co/Linly-AI/Chinese-LLaMA-7B

[zhilonglu]

Ok, when I download the Chinese-LLaMA-7B ,then another error occurs, FileNotFoundError: [Errno 2] No such file or directory: '/home/pretrain_models/Chinese-LLaMA-7B/pytorch_model.bin.index.json'

[fengyh3]

i guess your --pretrained_model_path is not right, it should be set to specific .bin file, which is xxx/chinese_llama_7b.bin

[zhilonglu]

Yeah, I can run it ,thanks a lot. Actually, i want to know how many GPU and CPU memories occupied by Chinese-LLaMA-7B mode when use deepspeed.

[nl8590687]

I also meet the same question. I used the model file that @fengyh3 said but get the same error too. The model file name is chinese_llama_13b.bin