LLaVA-NeXT icon indicating copy to clipboard operation
LLaVA-NeXT copied to clipboard

Published 20 hours ago • verified publisher icon LLaVA-VL

[Question] Is it possible to convert `LlavaMistralForCausalLM` to `LlavaNextForConditionalGeneration`?

Open kouyakamada opened this issue 5 months ago • 0 comments

This script(scripts/train/finetune_siglip_a4.sh) was used to train Mistal-v0.3 as a vision model. The training seemed to work and the model was output, but the model's architecture is LlavaMistralForCausalLM, so it cannot be inferred by transformers or vllm. Is there any way to convert this model to LlavaNextForConditionalGeneration or is there an alternative?

kouyakamada avatar Sep 01 '24 05:09 kouyakamada