[Model] Add new model provider: AI/ML API
From https://github.com/open-compass/VLMEvalKit/issues/1073
This PR adds first-class support for AI/ML API as a new unified model provider to VLMEvalKit. AI/ML API provides access to 300+ models, including GPT-4o, Claude 3.5, Gemini 1.5, DeepSeek, LLaMA3, Mistral, and many others — via a single OpenAI-compatible interface.
🔧 What was added
- ✅ New model wrapper:
vlmeval/api/aimlapi.py - ✅ Registration in
vlmeval/api/__init__.py - ✅ Multiple preconfigured model entries in
vlmeval/config.pyviaai_ml_api_models - ✅ Extended
api_modelswith support for AIMLAPI - ✅ English and Chinese documentation in
docs/en/Quickstart.mdanddocs/zh-CN/Quickstart.md - ✅ Ready-to-use
AIML_API_KEYenvironment variable support
🚀 Usage Example
AIML_API_KEY=your_key_here python run.py --data CCBench --model aimlapi_gpt_4o
✅ Checklist
- [x] Code follows repo structure
- [x] Docs (EN + ZH) updated
- [x] Tested on:
MMBench,CCBench,MME - [x] Uses OpenAI-compatible API structure
Let me know if you'd like to adjust or split this PR into parts.
Hi, @D1m7asis
Would you please share me a temporary account on the AI / ML API platform for testing (w. $10 credit, e.g.)? My email is [email protected].
@kennymckormick hi! Can i get a review here pls? Thanks!
Hi @kennymckormick - any chance we could get a review here? We'll implement changes within a day if anything needs to be adjusted