vllm-project
[RFC]: New eviction strategy for prefix cache indexer
Summary
This RFC mainly aims to introduce new eviction policy to enhance the robustness of the prefix cache routing system.
Motivation
The current prefix cache indexer uses a periodic eviction strategy, and the default time is 60min, the memory usage will gradually increase over time, which may cause OOM in large-scale scenarios (I haven't encountered it so far, just a guess based on the code). It is necessary to introduce some advanced strategies to control the size of the memory used.
Proposed Change
- Introduce a new interface in
aibrix/pkg/plugins/gateway/evictorto keep it extensible - Make eviction strategy as configurable
- Implement the earliest periodic eviction strategy
- Implement the LRU eviction strategy and take it as default eviction strategy
Alternatives Considered
No response
Capacity based is eviction would be helpful for OOM issues. Thanks for bringing this up. We plan to refactor the prefix cache strategy in next two week. If you have interest to make some contribution, that would be perfect as well!
@Jeffwan Sure, I'm interested in implementing new eviction strategy, could you assign it to me