[Question] How to decrease my loss?

[ifshine]

Hello! I use LIMA to train llama2-7b based on adapter.py. Below are my lr curve and loss curve. It's sad that the loss doesn't decrease. Could you offer guidance, techniques, or advice on decreasing the loss? Your experiences and insights are highly appreciated!🙏🙏🙏

[carmocca]

cc @rasbt in case you have tried this combination

[rasbt]

@carmocca Sorry, I only used LoRA recently. One thing to try for experimentation purposes is to truncate LIMA, i.e, preparing it with --max_seq_len 1028 so it's more comparable to Alpaca in case the hyperparams are an issue.

[ifshine]

@rasbt I wonder if it is convient for you to share your loss curve. Thanks!

[rasbt]

sry, I would have to rerun it again but plan to do so. Probably early next week! I'll make a reminder to give an update

[rasbt]

I am getting similarly noise loss curves but the model performance with LIMA is actually quite good despite this. Maybe the examples in LIMA are so different that it's normal to have such a noisy loss there.

Maybe it's also related to the sparse and noisy length distribution in LIMA:

But for reference, the Llama 2 model performance here (Alpaca converges quite well to ~0.5 loss but the model performance of LIMA, even though the loss fluctuates between 0.5 an 2.5 is actually better)

|   Model Name          | truthfulqa_mc1 | truthfulqa_mc2 | arithmetic_2ds | arithmetic_4da | blimp_causative | hendrycksTest-global_facts |
+-----------------------+----------------+----------------+----------------+----------------+-----------------+----------------------------+
| Llama 2 7B            |     0.2534     |     0.3967     |     0.508      |     0.637      |      0.787      |            0.32            |
+-----------------------+----------------+----------------+----------------+----------------+-----------------+----------------------------+
| 1 default LoRA Alpaca |     0.2876     |     0.4211     |     0.3555     |     0.0035     |       0.75      |            0.27            |
+-----------------------+----------------+----------------+----------------+----------------+-----------------+----------------------------+
| 2 default LoRA Alpaca |     0.284      |     0.4217     |     0.369      |     0.004      |      0.747      |            0.26            |
+-----------------------+----------------+----------------+----------------+----------------+-----------------+----------------------------+
| 3 default LoRA Alpaca |     0.2815     |     0.4206     |     0.372      |     0.004      |      0.747      |            0.27            |
+-----------------------+----------------+----------------+----------------+----------------+-----------------+----------------------------+
| 4 best LoRA Alpaca    |     0.3035     |     0.4664     |     0.8135     |     0.3025     |      0.746      |            0.32            |
+-----------------------+----------------+----------------+----------------+----------------+-----------------+----------------------------+
| 5 above setting w LIMA|     0.3072     |     0.4632     |     0.0245     |     0.0005     |      0.724      |            0.37            |
+-----------------------+----------------+----------------+----------------+----------------+-----------------+----------------------------+

Note that the above is for LoRA, not Adapter. Either I did something stupid or the evaluation script currently doesn't work with the Adapter checkpoints.

[Davegdd]

Hi @rasbt , were those results correct or was it actually some error on your part or the evaluation script? I have gotten something similar.