AssertionError: Padding_idx must be within num_embeddings

[zhaowt61]

Hi team, thanks for the amazing work! I got the following error when I tried to finetune InternVL-Chat-1.5:

[rank0]: Original Traceback (most recent call last): [rank0]: File "/opt/conda/envs/internvl_train2/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in _worker [rank0]: output = module(*input, **kwargs) [rank0]: File "/opt/conda/envs/internvl_train2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: File "/opt/conda/envs/internvl_train2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: File "/sensei-fs/users/wezhao/projects/VLM/InternVL/internvl_chat/internvl/model/internvl_chat/modeling_internvl_chat.py", line 125, in forward [rank0]: input_embeds = self.language_model.get_input_embeddings()(input_ids).clone() [rank0]: File "/opt/conda/envs/internvl_train2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl [rank0]: return self._call_impl(*args, **kwargs) [rank0]: File "/opt/conda/envs/internvl_train2/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl [rank0]: return forward_call(*args, **kwargs) [rank0]: File "/opt/conda/envs/internvl_train2/lib/python3.9/site-packages/torch/nn/modules/sparse.py", line 163, in forward [rank0]: return F.embedding( [rank0]: File "/opt/conda/envs/internvl_train2/lib/python3.9/site-packages/torch/nn/functional.py", line 2249, in embedding [rank0]: assert padding_idx < weight.size(0), "Padding_idx must be within num_embeddings" [rank0]: AssertionError: Padding_idx must be within num_embeddings