Benchmark and Evaluations, RL Alignment, Applications, and Challenges of Large Vision Language Models

A most Frontend Collection and survey of vision-language model papers, and models GitHub repository

Below we compile awesome papers and model and github repositories that

• State-of-the-Art VLMs Collection of newest to oldest VLMs (we'll keep updating new models and benchmarks).
• Evaluate VLM benchmarks and corresponding link to the works
• Post-training/Alignment Newest related work for VLM alignment including RL, sft.
• Applications applications of VLMs in embodied AI, robotics, etc.
• Contribute surveys, perspectives, and datasets on the above topics.

Welcome to contribute and discuss!

🤩 Papers marked with a ⭐️ are contributed by the maintainers of this repository. If you find them useful, we would greatly appreciate it if you could give the repository a star or cite our paper.

Table of Contents

• 📄 Paper Link/⛑️ Citation

• 1. 📚 SoTA VLMs
• 2. 🗂️ Dataset and Evaluation
  • 2.1. Large Scale Pre-Training & Post-Training Dataset
  • 2.2. Datasets and Evaluation for VLM
  • 2.3. Benchmark Datasets, Simulators and Generative Models for Embodied VLM

• 3. 🔥 Post-Training/Alignment/prompt engineering 🔥

  • 3.1. RL Alignment for VLM
  • 3.2. Regular finetuning (SFT)
  • 3.3. VLM Alignment Github
  • 3.4. Prompt Engineering
• 4. ⚒️ Applications
  • 4.1. Embodied VLM agents
  • 4.2. Generative Visual Media Applications
  • 4.3. Robotics and Embodied AI
    • 4.3.1. Manipulation
    • 4.3.2. Navigation
    • 4.3.3. Human-robot Interaction
    • 4.3.4. Autonomous Driving
  • 4.4. Human-Centered AI
    • 4.4.1. Web Agent
    • 4.4.2. Accessibility
    • 4.4.3. Healthcare
    • 4.4.4. Social Goodness
• 5. ⛑️ Challenges
  • 5.1. Hallucination
  • 5.2. Safety
  • 5.3. Fairness
  • 5.4. Alignment
    • 5.4.1. Multi-modality Alignment
      • 5.4.2. Commonsense and Physics Alignment
  • 5.5. Efficient Training and Fine-Tuning
  • 5.6. Scarce of High-quality Dataset

0. Citation

@InProceedings{Li_2025_CVPR,
    author    = {Li, Zongxia and Wu, Xiyang and Du, Hongyang and Liu, Fuxiao and Nghiem, Huy and Shi, Guangyao},
    title     = {A Survey of State of the Art Large Vision Language Models: Benchmark Evaluations and Challenges},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2025},
    pages     = {1587-1606}
}

1. 📚 SoTA VLMs

2. 🗂️ Benchmarks and Evaluation

2.1. Datasets for Training VLMs

2.2. Datasets and Evaluation for VLM

🧮 Visual Math (+ Visual Math Reasoning)

🎞️ Video Understanding

💬 Multimodal Conversation

🧠 Multimodal General Intelligence

🔎 Visual Reasoning / VQA (+ Multilingual & OCR)

📝 Visual Text / Document Understanding (+ Charts)

🌄 Text‑to‑Image Generation

🚨 Hallucination Detection / Control

2.3. Benchmark Datasets, Simulators, and Generative Models for Embodied VLM

3. ⚒️ Post-Training

3.1. RL Alignment for VLM

3.2. Finetuning for VLM

3.3. VLM Alignment github

3.4. Prompt Optimization

4. ⚒️ Applications

4.1 Embodied VLM Agents

4.2. Generative Visual Media Applications

4.3. Robotics and Embodied AI

4.3.1. Manipulation

4.3.2. Navigation

4.3.3. Human-robot Interaction

4.3.4. Autonomous Driving

4.4. Human-Centered AI

4.4.1. Web Agent

4.4.2. Accessibility

4.4.3. Healthcare

4.4.4. Social Goodness

5. Challenges

5.1 Hallucination

5.2 Safety

5.3 Fairness

5.4 Alignment

5.4.1 Multi-modality Alignment

5.4.2 Commonsense and Physics Alignment

5.5 Efficient Training and Fine-Tuning

5.6 Scarce of High-quality Dataset