swift icon indicating copy to clipboard operation
swift copied to clipboard

finetuning model with `max_length=4096`, but in infer got `exceeds the model max_length: 2048'

Open piqiuni opened this issue 2 months ago • 1 comments

Describe the bug What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程,最好有截图)

Finetuning model ModelType.qwen_vl_chat with max_length=4096 But in inference with the checkpoint, got exceedsthe model max_length: 2048 error

token len: history:421,  now: 1630

Traceback (most recent call last):
  File "/home/ldl/pi_code/swift/pi_code/infer_qwen_vl.py", line 83, in <module>
    response, _ = inference(model, template, value, history)
  File "/home/ldl/miniconda3/envs/swift/lib/python3.10/site-packages/swift/llm/utils/utils.py", line 748, in inference
    raise AssertionError('Current sentence length exceeds'
AssertionError: Current sentence length exceedsthe model max_length: 2048

Your hardware and system info Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息,如CUDA版本,系统,GPU型号和torch版本等)

Additional context Add any other context about the problem here(在这里补充其他信息)

piqiuni avatar May 02 '24 11:05 piqiuni

Seems like I can simply change the 'model.config.seq_length = 4096', and the output works fine.

piqiuni avatar May 02 '24 11:05 piqiuni