Rotary KV cache

[flaneur2020]

mistral's context window is longer than the kv cache with the help of sliding window attention. we can make kv cache a ring buffer, so that we can keep chat contiguously (although the model may forget the earlier things).