[Question] loss 不能下降

[awzhgw]

Question

When I fine-tune with Lora, the loss will drop to about 0.8 in the first 20k steps, and then increase to 6.2 over time, and then keep oscillating here. May I ask why this problem occurs? How to solve it? Translate this passage into English

[Sprinter1999]

Hi friend, I guess you may consider switching a lower learning rate when conducting your SFT.

[OliverLeeXZ]

Same question

[Vignesh-Valaboju]

Same problem too! Curious to know if lower learning rate helped you

[OliverLeeXZ]

I change learning rate from 2e-4 to 2e-5. It is really work!