aibrix
aibrix copied to clipboard
Implement fairness solutions like VTC in AIBrix
🚀 Feature Description and Motivation
Achieving efficient online LLM inference with SLO guarantees necessitates isolation among different clients is super important. Beside OSDI'24 VTC, I did see some new papers published in this area. Let's start to implement the fairness part in the solution.
- VTC https://www.usenix.org/conference/osdi24/presentation/sheng
- D2LPM https://arxiv.org/pdf/2501.14312
I talked with YIchuan in person from UC Berkeley retreat, if we need help from them, I can connect the efforts.
Use Case
No response
Proposed Solution
No response