Sanketh K S

icon indicating an email [email protected]

icon company Synamedia icon location Bangalore, India icon location DevOps, Cloud engineer, Data science, Python, build end-to-end systems

Results 1 comments of Sanketh K S

I want to close kv cache. if i set gpu_memory_utilization is 0. Does it means that i close the kv cache?

@WoosukKwon , Can we clear the KV cache in the GPU after prompting? I see that the GPU usage keeps increasing with every prompting. Is there any other solution for...