vllm-project
Pluggable Gateway
🚀 Feature Description and Motivation
I am a kgateway maintainer. In the upcoming release, kgateway will support Kubernetes Gateway API Inference Extension. The aibrix Gateway overlaps with Gateway API Inference Extension, specifically the endpoint picker extension. Can the aibrix Gateway component become extensible in the architecture? A spec, conformance tests, etc. can be defined to support aibrix Gateway extensibility. By doing so, aibrix will benefit from wider adoption.
Use Case
As a kgateway user with Gateway API inference Extension, I prefer to integrate with Aibrix instead of having separate Gateways.
Proposed Solution
Develop an interface, spec, conformance tests, etc. to make the aibrix extensible and allow integration with Gateway API inference extension conformant implementations.
Agreed here! We worked closely with aibrix contributors before and would love to continue that trend!
@Jeffwan @varungup90 I've looked for an aibrix community meeting in your docs and couldnt find one. Would be happy to pop in to to discuss what gaps we think need to be covered for GIE to be more composable with aibrix. If that sounds good, I can get a doc drafted on the topic.
Thanks! Looking forward to it!
+1 as an Istio and kgateway maintainer, I would like our users stay with whatever gateways they feel comfortable.
I followed the design of GIE in early days, and asked one question like If inference platform has their own model object, will there a conceptual conflict exists?, because according to the design, GIE has its own inferenceModel and inferencePool, but seems no answer. Nowadays, we met the question in our inference platform so we may consider to employ the envoy gateway ourself just because we don't want to have too much model-like objects to our users, which is confusing, and aibrix may introduce the same API as well, see https://github.com/vllm-project/aibrix/pull/299.
TBH, we don't want to reimplement the wheel. Not represent the aibrix community, but our platform.
For AIBrix, our goal is to not restrict users to use specific component or module and be as much extensible. We should schedule a meeting to discuss more on this topic.
@varungup90 Any further updates on this? We're evaluating whether we can bring Istio
@varungup90 - will this cause issues? We don't want to use another gateway and cause problems on our system when using this!
@varungup90 - will this cause issues? We don't want to use another gateway and cause problems on our system when using this!
Current gateway will continue to work as-is, it will be an additional gateway option to select from.