Meng Zhu

[V1][Core] Support offloading KV cache to CPU.

3

## TL;DR In V1, swap GPU KV cache blocks to CPU upon eviction and swap them back if there's a cache hit. ## Swap Strategy CPU → GPU swap-in happens...

[RFC]: Offload KV cache to CPU in V1

5

### Motivation. Offloading device KV cache to the CPU can be worthwhile if the transfer overhead outweighs the re-computation, saving precious GPU cycles. This is especially useful in cases such...