Woosuk Kwon

icon indicating a website link https://woosuk.me icon indicating an email [email protected]

icon company University of California, Berkeley icon location Berkeley, CA icon location CS PhD student at UC Berkeley, building @vllm-project

Results 284 comments of Woosuk Kwon

[Build] Cython compilation support fix

@gshtras Can you please provide a concrete number (how much improvements)?

[Core][ROCm][AMD] Add optional torchrun multi GPU executor

@zhuohan123 Could you please take a look at this? This PR is pretty interesting.

[Core][ROCm][AMD] Add optional torchrun multi GPU executor

@njhill I believe this can be related to #3466 ? Could you also take a look?

[RFC]: BatchLLM for better shared prefix utilizing in offline scenarios

Hi @xinji1 Thanks for the detailed RFC and the PR, and sorry for the late reply. If I understand correctly, the speedup comes from: 1. Sorting the requests in the...

GPU Model Runner V2

yeah let me fix the error The PR passed the pre-commit test in CI somehow.

[Bugfix][V1] Handle MLA in kv_cache_interface

@tlrmchlsmth Just wondering: Do you have any accuracy or performance benchmark numbers?

[TPU] Use Ray for default distributed backend

@njhill Good question. Actually, the MP backend would also work for TPUs. However, I think users such as GKE prefer Ray, because 1) they are interested in multi-host inference (which...

[Hardware][Intel CPU]use ipex varlen attention to compute prompts for better performance.

@jikunshang It seems like using IPEX somehow changes the output of the models and thus fails the model tests. Could you please take a look?

[V1][Metrics] Add API for accessing in-memory Prometheus metrics

@markmc Is this PR waiting for review? Or is it in progress?

[V1][Metrics] Add API for accessing in-memory Prometheus metrics

@markmc Can you please merge from main again?