Nikos Karampatziakis

icon indicating a website link http://lowrank.net/nikos icon indicating an email [email protected]

icon company Microsoft

Results 14 comments of Nikos Karampatziakis

KV cache with CPU offloading

I have started the PR, linked above. Also added the possibility to specify `cache_implementation="offloaded"` in `GenerationConfig`

KV cache with CPU offloading

> Curious to hear @n17s thoughts. @gerbenvv - These are all valid improvements. - A static offloaded cache is a great addition. - Have you done any benchmarks? - In...

KV cache with CPU offloading

Also, I suggest starting a new issue and link/mention this issue. When my PR is merged, this issue will be closed. And edit your PR to be fixing the issue...

KV cache with CPU offloading

> I have not subclassed from StaticCache since it doesn't share any of the code with it Understood. Subclassing is mostly for making code such as `if isinstance(foo, StaticCache):` work...