VLMEvalKit icon indicating copy to clipboard operation
VLMEvalKit copied to clipboard

[Model] Add new model provider: AI/ML API

Open D1m7asis opened this issue 6 months ago • 3 comments

From https://github.com/open-compass/VLMEvalKit/issues/1073

This PR adds first-class support for AI/ML API as a new unified model provider to VLMEvalKit. AI/ML API provides access to 300+ models, including GPT-4o, Claude 3.5, Gemini 1.5, DeepSeek, LLaMA3, Mistral, and many others — via a single OpenAI-compatible interface.


🔧 What was added

  • ✅ New model wrapper: vlmeval/api/aimlapi.py
  • ✅ Registration in vlmeval/api/__init__.py
  • ✅ Multiple preconfigured model entries in vlmeval/config.py via ai_ml_api_models
  • ✅ Extended api_models with support for AIMLAPI
  • ✅ English and Chinese documentation in docs/en/Quickstart.md and docs/zh-CN/Quickstart.md
  • ✅ Ready-to-use AIML_API_KEY environment variable support

🚀 Usage Example

AIML_API_KEY=your_key_here python run.py --data CCBench --model aimlapi_gpt_4o

✅ Checklist

  • [x] Code follows repo structure
  • [x] Docs (EN + ZH) updated
  • [x] Tested on: MMBench, CCBench, MME
  • [x] Uses OpenAI-compatible API structure

Let me know if you'd like to adjust or split this PR into parts.

D1m7asis avatar Jun 19 '25 22:06 D1m7asis

Hi, @D1m7asis

Would you please share me a temporary account on the AI / ML API platform for testing (w. $10 credit, e.g.)? My email is [email protected].

kennymckormick avatar Jun 26 '25 08:06 kennymckormick

@kennymckormick hi! Can i get a review here pls? Thanks!

D1m7asis avatar Aug 12 '25 08:08 D1m7asis

Hi @kennymckormick - any chance we could get a review here? We'll implement changes within a day if anything needs to be adjusted

OctavianTheI avatar Sep 04 '25 09:09 OctavianTheI