SangBin Cho comments

Results 409 comments of


                                            SangBin Cho

[Core] Implementing disaggregated prefilling, and caching KV cache in CPU/disk/database.

@KuntaiDu this is ready to review?

[Serve][Streaming] Intermediate deployment throughput drops when serving concurrent users.

I will assign a core label.

[Serve][Streaming] Intermediate deployment throughput drops when serving concurrent users.

I will report the concrete progress by eod today

[Serve][Streaming] Intermediate deployment throughput drops when serving concurrent users.

https://github.com/ray-project/ray/pull/40498

[Serve][Streaming] Intermediate deployment throughput drops when serving concurrent users.

@jsdir @shixianc @valtab can you guys tell me more details about the setup? Is it same as the issue here? (you have an intermediate router that's just having async responses)?

[Serve][Streaming] Intermediate deployment throughput drops when serving concurrent users.

We made some progress in the master though the throughput is not as good as gRPC streaming. Btw, @edoakes do we automatically batch requests in the server layer? Maybe we...

[Serve][Streaming] Intermediate deployment throughput drops when serving concurrent users.

@alexeykudinkin should we close this?

[ADAG] Add visualization of compiled graphs

cc @kevin85421 can you take a look?

[core][compiled graphs] Add CPU-based NCCL communicator for development

who's going to take this task?

[RFC]: Improve guided decoding (logit_processor) APIs and performance.

> Can you elaborate on why you think placing the guided decoding parameters in the SamplingParams is a good idea? As I commented in https://github.com/vllm-project/vllm/pull/4130, I think they conceptually overlap...