lightllm icon indicating copy to clipboard operation
lightllm copied to clipboard

feat: Support decode chunk PD serving mode

Open zhhangBian opened this issue 5 months ago • 1 comments

  • Add a new arg pd_chunk_size to decide the chunk size. 0 means no chunk
  • Support decode chunk

zhhangBian avatar Jun 25 '25 01:06 zhhangBian

I will remove the unnecessary files and changes later if the code review is all right.

The core changes is in the following two files:

  • lightllm/server/api_cli.py
  • lightllm/server/httpserver_for_pd_master/manager.py

zhhangBian avatar Jun 25 '25 01:06 zhhangBian