Loss spike during training phase

[yzxyzh]

Hi!

I'm trying to experiment you lora on different datasets. I observe that the loss is generally decreasing for each test I am performing, but also contain strange spikes. The loss value and decrease pattern is very similar with full-parameter fine-tune, thus I assume the final result should be fine, but those spikes are unexplainable to me. Similar spike has never been observed during full parameter fine-tuning. Any idea what has caused these spikes?

the spikes are marked with red boxes.

Thanks

[baibaiw5]

Hi! I meet the same problem

[Maxwell-Lyu]

Same here, but the trained model seems fine? (sort of)

[eggqq007]

because the qloar.py consume group_by_length.......

[artidoro]

Yes, group_by_length batches examples with similar length together and creates the oscillating pattern in the loss that you observe. The batches with shorter examples and those with longer ones have different loss values as they tend to correlate with "difficulty".

Using this setting is not necessary but will improve finetuning efficiency by reducing padding.