SangBin Cho comments

Results 292 comments of


                                            SangBin Cho

[Bugfix][Kernel] allow non-power-of-2 for prefix prefill with alibi

sorry I am going to take a look at it today!

[Bugfix][Kernel] allow non-power-of-2 for prefix prefill with alibi

can you try merge the latest master? I saw it sometimes happens... not sure what's the root cause

[3/N] Refactor scheduler for chunked prefill scheduling

Thanks for the quality review! I will finish addressing comments by eod today :)!

[3/N] Refactor scheduler for chunked prefill scheduling

Update: Swapping is currently only for decoding, so I think decoupling should not be a scope of this PR. But I can address this better in the next PR

[3/N] Refactor scheduler for chunked prefill scheduling

Here's the e2e PR for scheduler on top of this branch; https://github.com/rkooo567/vllm/pull/15

[3/N] Refactor scheduler for chunked prefill scheduling

Addressed comments relevant to this PR https://github.com/rkooo567/vllm/pull/15#issuecomment-2021809284. cc @simon-mo to take another look!

[3/N] Refactor scheduler for chunked prefill scheduling

Status: will separate out swapping and add a lot more unit tests.

[3/N] Refactor scheduler for chunked prefill scheduling

before: Throughput: 2.01 requests/s, 972.94 tokens/s after: Throughput: 1.99 requests/s, 961.40 tokens/s Benchmark result. I'd say it is just the same

[3/N] Refactor scheduler for chunked prefill scheduling

@simon-mo Updated (plz take a look one more time); - swap is a separate API now - each API is more thoroughly tested. Also better unit testing for swapping. -...

[3/N] Refactor scheduler for chunked prefill scheduling

@zhuohan123 @simon-mo As we discussed offline, I updated code based on the proposal I made. - Made all _schedule APIs as stateless as possible. - Each APIs can be used...