lightllm

lightllm copied to clipboard

Hello, I want to deploy the VILA model for serving VILA1.5-3B-AWQ (https://github.com/NVlabs/VILA). Could you please guide me on how to get started? Are there any specific instructions or tools I...

anhnhust

支持encoder-only模型

1

您好，请问您是否考虑支持encoder-only模型？如：bge-m3 embedding model等

EvanSong77

CPU Inference

1

Could lightllm use only CPU for inference?

JocelynPanPan

add chat template

1

shihaobai

add fp8_scaled_mm_per_token

1

WANDY666

[feature] Add prefix_kv_cache transfer between dp rankers.

1

WANDY666

feat: disk cache v1.0

1

blueswhen

add visual_send_bs args

1

SangChengC

add flashinfer fused_norm_add op

1

SangChengC

support Deepseek3.2

1

sufubao