GeoChat icon indicating copy to clipboard operation
GeoChat copied to clipboard
verified publisher icon mbzuai-oryx

Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([1297, 1024])), this look incorrect.

Open zzingok opened this issue 1 year ago โ€ข 9 comments

Loading checkpoint shards: 50%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ | 1/2 [00:03<00:03, 3.53s/it] Traceback (most recent call last): File "/home/zzj/GeoChat/geochat_demo.py", line 54, in tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name, args.load_8bit, args.load_4bit, device=args.device) File "/home/zzj/GeoChat/geochat/model/builder.py", line 104, in load_pretrained_model model = GeoChatLlamaForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, **kwargs) File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2903, in from_pretrained ) = cls._load_pretrained_model( File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3260, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/transformers/modeling_utils.py", line 717, in _load_state_dict_into_meta_model set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs) File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 358, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([1297, 1024])), this look incorrect.

zzingok avatar Aug 11 '24 11:08 zzingok

Same too.

JimmyMa99 avatar Sep 06 '24 08:09 JimmyMa99

I have been solved the problem, but I think it is not the best way. Maybe you need to add ignore_mismatched_sizes=True, in geochat/train/train.py#L802

then you will get the next error

RuntimeError: Error(s) in loading state_dict for Sequential:
        Missing key(s) in state_dict: "0.weight", "0.bias", "2.weight", "2.bias".

JimmyMa99 avatar Sep 06 '24 08:09 JimmyMa99

hello, do you fixed this error?

hshdbbxjwi avatar Sep 18 '24 14:09 hshdbbxjwi

hello, do you fixed this error?

Hey, I think it is necessary to use the same checkpoint to solve this problem. I try to extract the projector from the finally checkpoint, then it work. But I found that author say they use a different VIT, you can see here.

Hope that helps.

JimmyMa99 avatar Sep 19 '24 01:09 JimmyMa99

Loading checkpoint shards: 50%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ | 1/2 [00:03<00:03, 3.53s/it] Traceback (most recent call last): File "/home/zzj/GeoChat/geochat_demo.py", line 54, in tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name, args.load_8bit, args.load_4bit, device=args.device) File "/home/zzj/GeoChat/geochat/model/builder.py", line 104, in load_pretrained_model model = GeoChatLlamaForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, **kwargs) File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2903, in from_pretrained ) = cls._load_pretrained_model( File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3260, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/transformers/modeling_utils.py", line 717, in _load_state_dict_into_meta_model set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs) File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 358, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([1297, 1024])), this look incorrect.

Same too. Do you have a good solution?

ZhanYang-nwpu avatar Sep 20 '24 01:09 ZhanYang-nwpu

hello, do you fixed this error?

Hey, I think it is necessary to use the same checkpoint to solve this problem. I try to extract the projector from the finally checkpoint, then it work. But I found that author say they use a different VIT, you can see here.

Hope that helps.

Thank you

hshdbbxjwi avatar Sep 20 '24 01:09 hshdbbxjwi

Loading checkpoint shards: 50%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ | 1/2 [00:03<00:03, 3.53s/it] Traceback (most recent call last): File "/home/zzj/GeoChat/geochat_demo.py", line 54, in tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name, args.load_8bit, args.load_4bit, device=args.device) File "/home/zzj/GeoChat/geochat/model/builder.py", line 104, in load_pretrained_model model = GeoChatLlamaForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, **kwargs) File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2903, in from_pretrained ) = cls._load_pretrained_model( File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3260, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/transformers/modeling_utils.py", line 717, in _load_state_dict_into_meta_model set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs) File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 358, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([1297, 1024])), this look incorrect.

Same too. Do you have a good solution?

ๅปบ่ฎฎไฝ ๅƒreadme.mdไธญ่ฏด็š„้‚ฃๆ ท๏ผŒๅ…ˆgit pull ๏ผŒ็„ถๅŽๆŠŠไปŽhuggingfaceไธ‹่ฝฝ็š„ๆƒ้‡ๆ–‡ไปถๆ”พๅœจ็ฑปไผผweights/geochat่ฟ™ๆ ท็ป“ๆž„็š„ๆ–‡ไปถๅคนไธญ๏ผŒๅบ”่ฏฅๅฐฑๅฏไปฅไบ†

hshdbbxjwi avatar Sep 20 '24 01:09 hshdbbxjwi

Loading checkpoint shards: 50%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ | 1/2 [00:03<00:03, 3.53s/it] Traceback (most recent call last): File "/home/zzj/GeoChat/geochat_demo.py", line 54, in tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name, args.load_8bit, args.load_4bit, device=args.device) File "/home/zzj/GeoChat/geochat/model/builder.py", line 104, in load_pretrained_model model = GeoChatLlamaForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, **kwargs) File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2903, in from_pretrained ) = cls._load_pretrained_model( File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3260, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/transformers/modeling_utils.py", line 717, in _load_state_dict_into_meta_model set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs) File "/home/ps/anaconda3/envs/geochat/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 358, in set_module_tensor_to_device raise ValueError( ValueError: Trying to set a tensor of shape torch.Size([577, 1024]) in "weight" (which has shape torch.Size([1297, 1024])), this look incorrect.

Same too. Do you have a good solution?

ๅปบ่ฎฎไฝ ๅƒreadme.mdไธญ่ฏด็š„้‚ฃๆ ท๏ผŒๅ…ˆgit pull ๏ผŒ็„ถๅŽๆŠŠไปŽhuggingfaceไธ‹่ฝฝ็š„ๆƒ้‡ๆ–‡ไปถๆ”พๅœจ็ฑปไผผweights/geochat่ฟ™ๆ ท็ป“ๆž„็š„ๆ–‡ไปถๅคนไธญ๏ผŒๅบ”่ฏฅๅฐฑๅฏไปฅไบ†

ๅฅ‡ๆ€ชไบ†๏ผŒๆˆ‘็กฎๅฎžๆ˜ฏๆŒ‰็…ง่ฟ™ไธชๆญฅ้ชค่ฟ›่กŒ็š„ใ€‚ๅชไธ่ฟ‡ๆˆ‘ๆƒ้‡ๅญ˜ๅ‚จๅœจ GeoChat-7B/ ่ทฏๅพ„

่ฟ˜ๆœ‰ไธ€ไธช้—ฎ้ข˜ๅฐฑๆ˜ฏๆˆ‘็š„ clip-vit-large-patch14-336 ๆƒ้‡ๆ˜ฏ็ฆป็บฟไธ‹่ฝฝๅฅฝไบ†๏ผŒ้‡ๆ–ฐๅฎšไน‰ไบ†self.vision_tower_name ่ฟ™ไธชๆƒ้‡่ทฏๅพ„๏ผŒ็›ดๆŽฅ็ฆป็บฟ load ็š„ใ€‚ๅฐฑๅชๆœ‰่ฟ™ไธชๆญฅ้ชคไธไธ€ๆ ท๏ผŒ่€Œๆบไปฃ็ ๅบ”่ฏฅๆ˜ฏ็›ดๆŽฅ่‡ชๅŠจๅœจ HF ๅœจ็บฟload็š„๏ผŒ่ฏฅไธไผšๆ˜ฏ่ฟ™ไธชๅฏผ่‡ด็š„้—ฎ้ข˜ๅง๏ผŸ

ZhanYang-nwpu avatar Sep 20 '24 02:09 ZhanYang-nwpu

Same too.

Hello, could you tell me how to specifically resolve this error?

bubblebrow avatar Oct 17 '24 15:10 bubblebrow