Inferencing Finetuned model

[rfoxes]

trafficstars

Hello, I finetuned LLaVA onevision with Qwen2-7B. In the finetuning script, I set it to finetune just the adapter. When I am trying to inference my model, I am using the example code given in the repo, and I pass in the pretrained model with no model_base, and the name as 'llava_qwen' to the load_pretrained_model function. However, I keep getting this error:

OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory ../checkpoints/llavanext-Qwen-7B-siglip-so400m-patch14-384-onevision-20K.

I am not sure how to fix this. Can anyone help with this error?

[rfoxes]

Hello, I was able to fix the error by downloading the safe tensors from the huggingface repo, but now the output of the model for an image is ['!!!!!!!!!!!']. Is this an error that occurred when fine-tuning the model, or something wrong with inferencing it?

[sc268]

got the same output ['!!!!'] @rfoxes any suggestions? Thanks!

[rfoxes]

I downloaded the projector files, and it changed the output, but it is still unreadable.

Here is an example ["ADM<\u5c42\u51fa<\u5c42\u51fa<\u5c42\u51fa<\u5c42\u51fa<oppable<\u6cbb\u5b890\u5b9a\u5f8b (licted \u59f9 - xhttp<\u5c42\u51fa<\u5c42\u51fa<\u5c42\u51fa<\u5c42\u51fa<\u6cbb\u5b890\u5b9a\u5f8b<",-<",-<"class<",-<)){\n\n< multer<",-<",-<.Physics<.Physics<",-<",-<",-<\u9648\u5217<\u9648\u5217<",-<",-<chnitt\u4ea6 ++;\r\n\r\n (\u781c<..."]