vllm-project
Support high availability of gateway server for production users
🚀 Feature Description and Motivation
Envoy gateway is the most important data plane component in AIBrix, to ensure high availability (HA) for Envoy Gateway with an external process, we need to implement best practices for resiliency, fault tolerance, and load distribution in Kubernetes.
There're few things missing
- PDB: Consider to leverage PDB to prevent all envoy instances from being drained simultaneously.
- Probs: Ensure instances are only routed traffic when healthy.
- Node Spreading: Distribute instances across different nodes to avoid single-node failures.
- Graceful Shutdown Handling: Ensure external process exits cleanly during pod restarts or termination.
/cc @gaocegege mentioned this requirement earlier. I think we at least need to improve the setting first. stateful sync up could be a separate issue.
Use Case
HA setup for router
Proposed Solution
No response
@Jeffwan would also be nice to be able to schedule gateway (and control plane) on separate nodes from core GPU nodes running the vLLM pods because gateway workloads are largely static while GPU workloads may be dynamic
