Woosuk Kwon

icon indicating a website link https://woosuk.me icon indicating an email [email protected]

icon company University of California, Berkeley icon location Berkeley, CA icon location CS PhD student at UC Berkeley, building @vllm-project

Results 284 comments of Woosuk Kwon

[V1] Get input tokens from scheduler

cc @comaniac This PR seems to work correctly when using TP (or single GPU), but PP still generates gibberish outputs.

[V1] Get input tokens from scheduler

@comaniac @njhill @LiuXiaoxuanPKU I've update the PR with some simplification for spec decoding. PTAL.

[V1][Spec decode] Move drafter to model runner

cc @LiuXiaoxuanPKU

[Misc] Use torch.compile for basic custom ops

cc @youkaichao @bigPYJ1151 @bnellnm

[Misc] Use torch.compile for basic custom ops

@hmellor Thanks for bringing this up again. Yeah I think we don't need this PR anymore since our torch.compile integration is more mature now. I think we could turn it...

[V1][core] Support allowed_token_ids in V1 sampler

@catherinelee274 Closing this as #13210 is merged. Thanks!

[Bug]: v0.5.5 crash: "AssertionError: expected running sequences"

@zoltan-fedor Thanks for reporting the bug. Could you please use without `--num-scheduler-steps 8`? I think there were several bug fixes on it after v0.5.5.

[wip] scaled_mm_bw_sm100

@tylertitsworth Can you please take a look?

Reshape cache to be XQA kernel compatible

👀

[Build] Cython compilation support fix

@gshtras @Alexei-V-Ivanov-AMD Thanks for the PR. Do you have any performance numbers before & after this PR?