Jiaxin Shan
Jiaxin Shan
https://github.com/vllm-project/aibrix/issues/1677
Capacity based is eviction would be helpful for OOM issues. Thanks for bringing this up. We plan to refactor the prefix cache strategy in next two week. If you have...
@vie-serendipity thanks! I've finished the assignment
Remove from v0.2.0 release and move to v0.3.0.
I will consider to rewrite this issue. Seems the original proposed idea is not very close to what we need at this moment now.
@omerap12 We made some improvements to vLLM here https://github.com/vllm-project/aibrix/pull/1429 I am not 100% sure if that's related. could you build a new image and update the router? Please also try...
@omerap12 HTTPRoute configuration doesn't help in this case. P/D router directly talk to the pod and bypass HTTPRoute. could you try to bump aibrix-gateway plugin to v0.4.1 and give another...
@omerap12 I will spend some time this weekend to preproduce the problem on EKS.
it's the manifest issue, v0.2.0-rc.2 seems are not well cut. As we can v0.2.0 now, we do not need to worry about the RC version
We have an initial PR in v0.2.0 release, we should work on stablization in v0.3.0.