DefTruth

icon indicating a website link https://github.com/xlite-dev

icon company @xlite-dev, @vipshop icon location Guangzhou, China icon location @xlite-dev, @vipshop, LeetCUDA.

Results 214 comments of DefTruth

Add ONNX Export and MNN/TNN/ONNXRuntime C++ Demo

@yucornetto hi~ would you like to review this PR?

[Bug]: Qwen1.5-72B L20x8 latest vLLM TPOT slower than v0.4.0.post, 48ms vs 39ms, why?

maybe relate to https://github.com/vllm-project/vllm/pull/5207

[Bug]: Qwen1.5-72B L20x8 latest vLLM TPOT slower than v0.4.0.post, 48ms vs 39ms, why?

@youkaichao close, seems the latest vllm (up to #5410) has fixed this problem. (TP0T 45ms v0.4.2 -> 39ms v0.5, eager mode) ```bash [I][2024-06-11 16:31:36][ 1/20][ 1/20 5%] session:0 turn:0 req:0...

[tensorrt-llm backend] A question about launch_triton_server.py

> In tensorrt_llm_backend, when we launch several server by MPI with world_size > 1, only the rank 0 (main process) will recieve/return requests. Other ranks will skip this step and...