[Bug] 更改session_len无法解决输出截断的问题

[Amber-Believe]

Checklist

• [x] 1. I have searched related issues but cannot get the expected help.
• [x] 2. The bug has not been fixed in the latest version.
• [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

（多模态qwen2.5-vl-32b推理）要求输出的结果token比较长，使用默认参数会发生输出内容被截断的情况，更改session_len还是无法解决问题，还是会出现截断，我该怎么能增加输出token数呢？

Reproduction

with pipeline('/data02/wangq-bj/wq/LLaMA-Factory-main/output/Bridge_layout/20250413_2230_qwen2.5_vl-32b_lora_sft', backend_config=TurbomindEngineConfig(tp=1, session_len=8192,)) as pipe: image = load_image('/data02/wangq-bj/download/data/立面/0_测试_1/单页/314-S346 报批稿PDF汇总/314-S346 报批稿PDF汇总_04 第四篇 桥梁涵洞#P8.png') response = pipe(("\n输出该图的图类型、桥梁中心桩号、起点桩号、终点桩号、左右幅信息，输出图中所有的墩台号，输出有盖梁、承台、空心墩柱、柱系梁、桩系梁的墩台号，输出跨径、上部结构形式、伸缩缝规格、搭板长度、桥台、柱高、柱径、桩高、桩径、剖面图位置的具体内容以及对应的墩台号。\n 以下是任务指令：\n 1.将信息提取的结果以 json 格式返回，不要输出多余信息。\n 2.无法提取到的信息以空字典{} 代替。\n 以下是输出格式示范，禁止输出其他多余信息：\n {\n "图类型": "立面",\n "桥梁中心桩号": "K1+817",\n "起点桩号": "K1+633",\n "终点桩号": "K2151+197.09",\n "左右幅": "左幅",\n\n "墩台号": [0,1,2,3],\n "承台": [2,3],\n "盖梁": [1,2],\n "空心墩柱": [2],\n "桩系梁": [],\n "柱系梁": [],\n\n "跨径": {"0-5": "53250=16250","5-7": "23250=6500"},\n "上部结构形式": {"0-5": "预应力混凝土简支（桥面连续）T梁","5-7": "预应力混凝土简支（桥面连续）T梁"},\n "伸缩缝规格": {'0':'D80','4':'D160'},\n "搭板长度": {'0':'5m'},\n "桥台":{'0':'肋板台'},\n "柱高": {"1": "130","2": "130","3": "130",},\n "柱径": {"1": "130","2": "130","3": "130",},\n "桩高": {"0": "4300","1": "4300","2": "4300","3": "4300",},\n "桩径": {"0": "150","1": "150","2": "150","3": "150",},\n "剖面图位置": { "4-5": "A",}}\n ", image), max_new_tokens = 1024,min_new_tokens = 1024) print(response.text)

Clear the torch cache and perform garbage collection if needed

import torch import gc torch.cuda.empty_cache() gc.collect()

Environment

sys.platform: linux
Python: 3.9.21 (main, Dec 11 2024, 16:24:11) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1,2,3,4,5,6,7: NVIDIA H100 80GB HBM3
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.1, V12.1.105
GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 2.6.0+cu124
PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 12.4
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 90.1
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=2236df1770800ffea5697b11b0bb0d910b2e59e1, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 

TorchVision: 0.21.0+cu124
LMDeploy: 0.7.3+
transformers: 4.49.0
gradio: 4.44.1
fastapi: 0.115.12
pydantic: 2.11.3
triton: 3.2.0
NVIDIA Topology: 
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    SYS     SYS     0-47,96-143     0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    NODE    NODE    SYS     SYS     0-47,96-143     0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    NODE    SYS     SYS     0-47,96-143     0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    PIX     SYS     SYS     0-47,96-143     0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     PIX     NODE    48-95,144-191   1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     NODE    PIX     48-95,144-191   1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     NODE    NODE    48-95,144-191   1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     NODE    NODE    48-95,144-191   1               N/A
NIC0    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    SYS     SYS                             
NIC1    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     NODE     X      SYS     SYS                             
NIC2    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    SYS     SYS      X      NODE                            
NIC3    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    SYS     SYS     NODE     X                              

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3

Error traceback

[CUHKSZzxy]

@Amber-Believe

可以参考以下代码，把 generation config 中的 max_new_tokens 参数调大。max_new_tokens 是控制输出长度上限的

import os
from lmdeploy import pipeline, PytorchEngineConfig, GenerationConfig
from lmdeploy.vl import load_image

os.environ['CUDA_VISIBLE_DEVICES'] = '7'

model_path = "xxx"

# Configure the PyTorch backend engine
pt_backend_config = PytorchEngineConfig(
    tp=1,
    cache_max_entry_count=0.8, 
    session_len=8192,  # Maximum session length (context length)
)


gen_config = GenerationConfig(
    max_new_tokens=8192,  # Maximum number of new tokens to generate
)

def main():
    # Initialize the pipeline with the model and backend configuration
    pipe = pipeline(
        model_path,
        backend_config=pt_backend_config,
    )

    image_url = 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg'
    image = load_image(image_url)
    prompt = "Describe this image."
    response = pipe((prompt, image), gen_config=gen_config)

    print(response)


if __name__ == '__main__':
    main()

[Amber-Believe]

好的 问题已解决

[zxy1728]

session_len

请问session_len的长度包含输出token吗？还是只对输入长度有效