LLaVA
LLaVA copied to clipboard
[Question] loss 不能下降
Question
When I fine-tune with Lora, the loss will drop to about 0.8 in the first 20k steps, and then increase to 6.2 over time, and then keep oscillating here. May I ask why this problem occurs? How to solve it? Translate this passage into English
Hi friend, I guess you may consider switching a lower learning rate when conducting your SFT.
Same question
Same problem too! Curious to know if lower learning rate helped you
I change learning rate from 2e-4 to 2e-5. It is really work!