LLaVA-VL
0.5b model work fine,7b model result is `['']`
I redownload this repo,and tried transfoemers version:4.40.0.dev、4.40.0、4.41.2,the result is still [''].
some thing i do include:
All weight i use is local weight.below is my change.
-
Meta-Llama-3-8B-Instruct:llava/conversation.py,line387, tokenizer=AutoTokenizer.from_pretrained("local_path/LLaVA-NeXT/Meta-Llama-3-8B-Instruct") -
siglip-so400m-patch14-384:llava-onevision-qwen2-7b-si/config.json,line176, ision_tower": "local_path/siglip-so400m-patch14-384",then some error about mismatch,I use this to fix it.https://github.com/LLaVA-VL/LLaVA-NeXT/issues/148#issuecomment-2298549964
then 0.5b model work fine,7b model result is always [''],below result is 7b model :
(llava) root@sugon:~/work/project/LLaVA-NeXT# python demo_single_image.py
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loaded LLaVA model: /root/work/project/LLaVA-NeXT_bak/llava-onevision-qwen2-7b-si
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
You are using a model of type llava to instantiate a model of type llava_qwen. This is not supported for all configurations of models and can yield
Loading vision tower: /root/work/project/LLaVA-NeXT/siglip-so400m-patch14-384
Loading checkpoint shards: 100%|___________________________________________________________________________________________________________________
Model Class: LlavaQwenForCausalLM
['']
how can i fix it.please give me some advice
for question 1, actually, you can modifie it like this: (1) tokenizer=None and: (2) if self.tokenizer is None: self.tokenizer = AutoTokenizer.from_pretrained(self.tokenizer_id)
print(chat_template_messages)
return self.tokenizer.apply_chat_template(chat_template_messages, tokenize=False, add_generation_prompt=True)
I found my problem because the weight file I downloaded was broken, and I fixed it by re-downloading the full weight file
The same issue. 7b model result is [!!.....] many many !!!
same here. have you fixed it?
The same issue. 7b model result is [!!.....] many many !!!
same here. have you fixed it?
Please see my latest reply.https://github.com/LLaVA-VL/LLaVA-NeXT/issues/186#issuecomment-2363249412
The same issue. 7b model result is [!!.....] many many !!!
same here. have you fixed it?
You can re-download the LLaMA checkpoint or check its version. The output became normal when I used this.
I found my problem because the weight file I downloaded was broken, and I fixed it by re-downloading the full weight file
How did you confirm that your weight file was broken? By size or something? Because I thought that if it is broken, it should raise error when loading it.
I found my problem because the weight file I downloaded was broken, and I fixed it by re-downloading the full weight file
How did you confirm that your weight file was broken? By size or something? Because I thought that if it is broken, it should raise error when loading it. It was quite difficult. At that time, I didn’t have any doubts, but later I found that many people suggested downloading it again. So I gave it a try, and it worked.
I found my problem because the weight file I downloaded was broken, and I fixed it by re-downloading the full weight file
How did you confirm that your weight file was broken? By size or something? Because I thought that if it is broken, it should raise error when loading it. It was quite difficult. At that time, I didn’t have any doubts, but later I found that many people suggested downloading it again. So I gave it a try, and it worked.
Well, I just upgraded pytorch from 2.1.0 to 2.3.1 and now the model replies normally. So I think the root cause is some precision related problem and has been fixed in torch-2.3.1 in my case. Thanks for answering!
I found my problem because the weight file I downloaded was broken, and I fixed it by re-downloading the full weight file
How did you confirm that your weight file was broken? By size or something? Because I thought that if it is broken, it should raise error when loading it. It was quite difficult. At that time, I didn’t have any doubts, but later I found that many people suggested downloading it again. So I gave it a try, and it worked.
Well, I just upgraded pytorch from 2.1.0 to 2.3.1 and now the model replies normally. So I think the root cause is some precision related problem and has been fixed in torch-2.3.1 in my case. Thanks for answering! Good discover.