Which file is the lora weight?

[hxx-who]

Hi, Great work! I was wondering what is the difference between different model files, like bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt, mp_rank_00_model_states.pt, pytorch_model.bin which is obtained after the zero_to_fp32.py, and the file obtained after merge_lora_weights_and_save_hf_model.py. Which of them is the lora weight? or they are just different formats of lora weight? Thanks!

[hxx-who]

and which of them is the full model weight?

[GaoXiaoshan]

+1

[GaoXiaoshan]

I have met the same problem. If you train with deepspeed, save the model and run python zero_to_fp32.py, you will get the pytorch_model.bin. The pytorch_model.bin is the full model weight which means there is no need to merge.