[BUG] <title>进行Lora微调时，训练数据集的每轮对话的图片输入是否有数量限制？

[L1NINE]

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

• [X] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

• [X] 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

在用一批驾驶场景相关的数据集（NuInstruct, NuscenesQA等，经过了格式的转换）进行微调时，遇到了这样的bug：

File "/root/.cache/huggingface/modules/transformers_modules/Qwen-VL-Chat/modeling_qwen.py", line 557, in forward
assert (bos_pos[0] == eos_pos[0]).all()
RuntimeError: The size of tensor a (8) must matchthe size of tensor b (7) at non-singleton dimension 0

我减少了每轮对话的输入图片数量（从三十多张减少到六张），把训练运行起来了，但对这个问题还是存在疑惑 有人试过用更多的图片进行训练吗？

期望行为 | Expected Behavior

No response

复现方法 | Steps To Reproduce

No response

运行环境 | Environment

- OS:
- Python:
- Transformers:
- PyTorch:2.1.2
- CUDA (`python -c 'import torch; print(torch.version.cuda)'`):

备注 | Anything else?

No response