lightllm icon indicating copy to clipboard operation
lightllm copied to clipboard
verified publisher icon ModelTC

Add fake balance for EP mode

Open STwangyingrui opened this issue 5 months ago • 0 comments

Add fake balance for EP mode, which is controled by option of --enable_ep_fake_balance. Cost: EP8 batch128 input64 (40+ different seqlens) totally cost about 5 seconds. Benefit: prefill throughput increase 35%, decoding throughput increase 15%, and the overheads become stable.

STwangyingrui avatar Jul 08 '25 07:07 STwangyingrui