about finetune_task_lora.sh

[simplelifetime]

Describe the issue

What does mm_projector_lr mean in the training script. Does it indicate that the mm_projector will be tuned during lora training? But I don't see the mm_projector parameters saved in the result checkpoint files. Can you show how to load the model after lora fine-tuning on custom tasks?

[haotian-liu]

it means that the learning rate of the mm_projector and it should be saved in non_lora_trainables.bin

[ZY123-GOOD]

hello. Do you know how to load the model after lora fine-tuning? I have encountered: Some weights of LlavaLlamaForCausalLM were not initialized from the model checkpoint at /home/yaozhu/LLaVA/LLaVA_codes/vicuna and are newly initialized: ['model.mm_projector.0.weight', 'model.mm_projector.2.bias', 'model.mm_projector.2.weight', 'model.mm_projector.0.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

[tosiyuki]

hello. Do you know how to load the model after lora fine-tuning? I have encountered: Some weights of LlavaLlamaForCausalLM were not initialized from the model checkpoint at /home/yaozhu/LLaVA/LLaVA_codes/vicuna and are newly initialized: ['model.mm_projector.0.weight', 'model.mm_projector.2.bias', 'model.mm_projector.2.weight', 'model.mm_projector.0.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

The following is how to LOAD mm_projector after tuning by LoRA. https://github.com/haotian-liu/LLaVA/blob/786aa6a19ea10edc6f574ad2e16276974e9aaa3a/llava/model/builder.py#L60-L75

[ZY123-GOOD]

Thanks!

[bluesky333]

I had this issue where finetune_task_lora.sh doesn't create mm_projector.bin which also limited my usage of the finetuned_lora model. (I cannot merge or use it for inference). I changed extract_mm_projector to extract mm_projector.bin from non_lora_trainables.bin.

[terminator123]

I had this issue where finetune_task_lora.sh doesn't create mm_projector.bin which also limited my usage of the finetuned_lora model. (I cannot merge or use it for inference). I changed extract_mm_projector to extract mm_projector.bin from non_lora_trainables.bin.

dou you known the non_lora_trainables.bin saves the merged mm_projector params or only the lora part of it ?

[cherry956]

I have my own dataset, now I want to use it to finetune LLava-v1.5-7b with lora. After finetune the model, I want to use it to predict. Can you tell me some detailed steps what should I do? I am contact it the first time. Thanks.

[mrseanryan]

@cherry956 in case it helps - this fork shows how to fine-tune v1.5

https://github.com/mrseanryan/finetune_LLaVA