Phi-3CookBook
Phi-3CookBook copied to clipboard
Wrong LoRA tuning example
Please provide us with the following information:
This issue is for a: (mark with an x
)
- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)
https://github.com/microsoft/Phi-3CookBook/blob/main/code/04.Finetuning/Phi-3-finetune-lora-python.ipynb https://github.com/microsoft/Phi-3CookBook/blob/main/code/04.Finetuning/Phi-3-finetune-qlora-python.ipynb
The example here in these example scripts appears to be misleading, as it in peft_config
, they show
target_modules = ['k_proj', 'q_proj', 'v_proj', 'o_proj', 'gate_proj', 'down_proj', 'up_proj']
,
which is incorrect based on the model's architecture.
It should be
target_modules = ['qkv_proj', 'o_proj', 'gate_up_proj', 'down_proj']
,
### Tasks