Woosuk Kwon comments

Results 284 comments of


                                            Woosuk Kwon

[Core] Move detokenization to front-end process

QQ: How does this work with #6913?

[V1][WIP] 2nd try of Hybrid allocator for full attention & sliding window attention interleaved models

Hi @heheda12345, thanks for the great work and sorry again for the delays in my review. This work is truly amazing! Now I think I (almost) fully understand the idea....

[V1] Implement sliding window attention in kv_cache_manager

@zhuohan123 Did you have a chance to take a look?

[V1] Implement sliding window attention in kv_cache_manager

As we discussed offline, I think we need a clear separation of the two APIs of `SpecializedManager`: 1. The first API describing how to free the KV cache for a...

[v1] AttentionMetadata for each layer

@heheda12345 Thanks! Please merge from main.

[v1] AttentionMetadata for each layer

@heheda12345 Please fix the lint error 😓

[v1] Pass BlockTable and KVCacheSpec to AttentionMetadataBuilders

> The precommit error is quite strange, I don't know how to fix it :( It happens on the main branch because of another PR. Please don't care about it.

[v1] Pass BlockTable and KVCacheSpec to AttentionMetadataBuilders

@heheda12345 Please fix the CI failure 😅

[V1][Core] Support offloading KV cache to CPU.

Hi @mengzhu28, thanks for submitting the great PR! I will reach out to you offline.

[V1][Core] Support offloading KV cache to CPU.

@mengzhu28 Could you please rebase the PR?