mistral.rs icon indicating copy to clipboard operation
mistral.rs copied to clipboard

Published 20 hours ago • verified publisher icon EricLBuehler

Any plan about KV compression algorithm like SnapKV and PyramidKV?

Open chenwanqq opened this issue 7 months ago • 4 comments

Hi, I'm wondering if you have any plans regarding kv compression methods like SnapKV and PyramidKV. These methods can reduce the use of memory for KV cache, hence improving availability on low-memory machines. Maybe I can make some contributions to this.

chenwanqq avatar Jul 20 '24 16:07 chenwanqq