vllm-project
[RFC]: Make API Gateway interface OpenAI compatible
Summary
This RFC proposes making the API Gateway interface within AIBrix compatible with OpenAI. We met few issues in the past few days. and @gaocegege also suggest https://github.com/vllm-project/aibrix/issues/732 earlier to document compatible APIs
Motivation
Currently, there are configurations within AIBrix that are not fully compatible with the OpenAI API. As a result, some OpenAI SDK may not sync with AIBrix deployments, which has been previously noted in issue #757. This lack of compatibility significantly impacts the adoption of AIBrix in scenarios where seamless integration with OpenAI-related ecosystems is desired.
Proposed Change
- Inference Interface Compatibility: Update the API Gateway code to be fully compatible with OpenAI-compatible format not only interface but also some headers, error response format etc..
- Operational Interface Compatibility with vLLM: LLM has its own set of operational requirements and interfaces, such as handling model loading etc. AIBrix's operational interface should be made to match these requirements.
- Extension of Metadata Service for New Interfaces: In cases where the Envoy gateway is not suitable, such as for batch APIs, the metadata service can be extended. The metadata service can store and provide additional information about the new interfaces, such as API version, input/output schemas, and usage limits.
- Building Test Tools with OpenAI SDK for Compatibility Verification: Develop test scripts using the OpenAI SDK that can send requests to AIBrix's API Gateway and verify the responses. These tests should cover all the supported OpenAI - compatible endpoints and edge cases. Replace curl based with sdk based queries.
- Documenting Supported Compatible APIs: Create a comprehensive document that details all the OpenAI - compatible APIs in AIBrix. This should include the API endpoints, request/response formats, authentication methods, and any limitations or caveats. tracking issue: #757
Alternatives Considered
No response
In v0.3.0, we will just make basic interface compatible and document the current situation. We may move some work to v0.4.0
I notice the error response format is completely incompatible. after checking https://github.com/openai/openai-openapi/tree/master?tab=readme-ov-file
- Gateway is missing required fields: type, param
- Gateway uses HTTP status code (int) for code instead of error code string
- 401 errors return invalid JSON: {"error":"unauthorized"}
- Error responses don't match OpenAI format. This affects: authentication errors, validation errors, rate limits, server errors, routing