[Question] pretain 7B loss doesn't change

[aprilehannibal]

Question

I tried to pretrain llava-7b on 8 * A100(40G), CUDA117, torch==1.13.1, py3.10 I found the loss doesn't change when training.

here is my pretraining scripts:

[aprilehannibal]

I wonder the LLM you used for pretraining is the "lmsys/vicuna-7b-delta-v1.1" or the origin "llama 7b" weight. @haotian-liu

[haotian-liu]

Hi @aprilehannibal, you need to perform weight conversion for vicuna weights, by combing llama-7b and vicuna-delta -> vicuna-7b. We pretrain based on vicuna-7b.

We'll update the instruction to make it clearer, thanks.

[aprilehannibal]

OK, got it! Thanks a lot!