lightllm icon indicating copy to clipboard operation
lightllm copied to clipboard

verified publisher icon ModelTC
→

Metadata

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

Results 125 lightllm issues
Sort by recently updated
recently updated
newest added

enable fa3 and fused_shared_experts by default

sufubao avatar sufubao

fix tl.where and set the default loader worker number

sufubao avatar sufubao

feat: Implementing Past Future Scheduler

WuSiYu avatar WuSiYu

Fp8 deepseek

blueswhen avatar blueswhen

Support Qwen models' dp>1 in PD

zhhangBian avatar zhhangBian

Asynchicache

jinbiaoyu avatar jinbiaoyu

where can I find `lightllm_constraint_decode_kernel`?

1 comment

The $\text{Pre}^3$ branch (https://github.com/ModelTC/LightLLM/tree/pre3-integrated) requires package `lightllm_constraint_decode_kernel` to run. However, I cannot find this package from the repositories in this organization. Is there any open-source implementation of $\text{Pre}^3$ that is...

Gompyn avatar Gompyn

Multimodal improve

shihaobai avatar shihaobai

add rmsnorm-add fusion kernel

theNiemand avatar theNiemand

cuda graph pool with LRU

STwangyingrui avatar STwangyingrui

← Metadata

2.3k
Stars
191
Forks
Watchers

Owner

verified publisher icon ModelTC

Metadata

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

Back