Phi-3CookBook icon indicating copy to clipboard operation
Phi-3CookBook copied to clipboard

Wrong LoRA tuning example

Open sofyc opened this issue 5 months ago • 0 comments

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

https://github.com/microsoft/Phi-3CookBook/blob/main/code/04.Finetuning/Phi-3-finetune-lora-python.ipynb https://github.com/microsoft/Phi-3CookBook/blob/main/code/04.Finetuning/Phi-3-finetune-qlora-python.ipynb

The example here in these example scripts appears to be misleading, as it in peft_config, they show

target_modules = ['k_proj', 'q_proj', 'v_proj', 'o_proj', 'gate_proj', 'down_proj', 'up_proj'],

which is incorrect based on the model's architecture. 366854892-e031d0bf-a871-40d7-b35f-8a04b56979e8

It should be target_modules = ['qkv_proj', 'o_proj', 'gate_up_proj', 'down_proj'],

### Tasks

sofyc avatar Sep 12 '24 22:09 sofyc