[Bug] Sizes of tensors must match except in dimension 0. Expected size 9 but got size 13 for tensor number 1 in the list

[BryantKai]

Checklist

• [ ] 1. I have searched related issues but cannot get the expected help.
• [ ] 2. The bug has not been fixed in the latest version.
• [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

26B的模型使用vllm推理的时候在某些图片上会报这个错

Reproduction

CUDA_VISIBLE_DEVICES=1,2 python -m vllm.entrypoints.openai.api_server --model /home/maasuser/InternVL2-26B --port 6781 --trust-remote-code --max_num_seqs 16 --max-model-len 4096 --chat-template /root/tmp/models/nvidia-ADPTV_InternVL2-8B/template/template.jinja --tensor-parallel-size 2 --gpu-memory-utilization

Environment

vllm A100 40GB显卡

Error traceback

No response

[alanayu]

This is a very serious problem. Hasn’t everyone encountered it?

[Weiyun1025]

Thank you for your interest in our work. It seems to be an issue about vLLM. However, I haven’t encountered this error when using vLLM. Could you provide more details?