What are the evaluation plans for each modality?

[bobo0810]

1. Image Understanding：the official VLMEvalKit library ✅
2. Multi-image and Video Understanding❓
3. Audio Understanding❓
4. Speech Generation❓
5. End-to-end Voice Cloning❓
6. Multimodal Live Streaming❓

[mjuicem]

Hi @bobo0810 ,

Thanks for your question! I will provide some details specifically for Multimodal Live Streaming, as you requested.

For evaluating MiniCPM-o 2.6's capabilities in Multimodal Live Streaming, you can refer to the following repository: https://github.com/THUNLP-MT/StreamingBench. This repository provides a comprehensive benchmark and evaluation framework for streaming MLLMs.

Here's a step-by-step guide to reproduce the results for MiniCPM-o 2.6 on the StreamingBench:

1. Inference Code: The inference code for MiniCPM-o 2.6 is located within the src/model directory of the StreamingBench repository.

2. Evaluation Pipeline: Follow the "Evaluation Pipeline" instructions in the StreamingBench repository's README. This involves three main stages:

   • Data Preparation
   • Model Preparation
   • Evaluation

By following these steps and the instructions within the StreamingBench repository, you should be able to fully reproduce the evaluation results for MiniCPM-o 2.6 on Multimodal Live Streaming. Let me know if you have any further questions!

[bobo0810]

@mjuicem Thank you very much. May I ask how to reproduce the indicators for multiple pictures, videos and audio?

[Cuiunbo]

@lihytotoro evaluation for multiple pictures, videos and audio

https://github.com/OpenBMB/UltraEval-Audio evaluation for audio