[Question] Question about LLaVA bench

[czczup]

Question

Hi, thanks for your awesome work. I have some question about the LLaVA Bench, which one is the score should I report in the paper?

[xiechengmude]

LM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).

• This IS NOT expected if you are initializing LlavaLlamaForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). 0%| | 0/1495 [00:00<?, ?it/s] Traceback (most recent call last): File "/root/miniconda3/envs/llava/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/root/miniconda3/envs/llava/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/workspace/LLaVA/llava/eval/model_vqa_qbench.py", line 122, in eval_model(args) File "/workspace/LLaVA/llava/eval/model_vqa_qbench.py", line 78, in eval_model image_tensor = image_processor.preprocess(image, return_tensors='pt')['pixel_values'].half().cuda() AttributeError: 'NoneType' object has no attribute 'preprocess'

[gapjialin]

LM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).

* This IS NOT expected if you are initializing LlavaLlamaForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  0%|                                                                                                                                                                 | 0/1495 [00:00<?, ?it/s]
  Traceback (most recent call last):
  File "/root/miniconda3/envs/llava/lib/python3.10/runpy.py", line 196, in _run_module_as_main
  return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/llava/lib/python3.10/runpy.py", line 86, in _run_code
  exec(code, run_globals)
  File "/workspace/LLaVA/llava/eval/model_vqa_qbench.py", line 122, in 
  eval_model(args)
  File "/workspace/LLaVA/llava/eval/model_vqa_qbench.py", line 78, in eval_model
  image_tensor = image_processor.preprocess(image, return_tensors='pt')['pixel_values'].half().cuda()
  AttributeError: 'NoneType' object has no attribute 'preprocess'

Hi, I also encountered the same problem. How did you solve it?

[daiqing98]

LM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).

* This IS NOT expected if you are initializing LlavaLlamaForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  0%|                                                                                                                                                                 | 0/1495 [00:00<?, ?it/s]
  Traceback (most recent call last):
  File "/root/miniconda3/envs/llava/lib/python3.10/runpy.py", line 196, in _run_module_as_main
  return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/llava/lib/python3.10/runpy.py", line 86, in _run_code
  exec(code, run_globals)
  File "/workspace/LLaVA/llava/eval/model_vqa_qbench.py", line 122, in 
  eval_model(args)
  File "/workspace/LLaVA/llava/eval/model_vqa_qbench.py", line 78, in eval_model
  image_tensor = image_processor.preprocess(image, return_tensors='pt')['pixel_values'].half().cuda()
  AttributeError: 'NoneType' object has no attribute 'preprocess'

Hi, I also encountered the same problem. How did you solve it?

Hi have you solved this problem? @gapjialin

[gaurav22verma]

@daiqing98 you might want to check #444 – there's a naming convention that should be followed

[Dominic789654]

@czczup Hi, can you tell me which score to report?