[Question] Slow Speed of vLLM when evaluating MMLU

[cby-pku]

Required prerequisites

• [X] I have read the documentation https://align-anything.readthedocs.io.
• [X] I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• [X] Consider asking first in a Discussion.

Questions

When evaluating MMLU, the codebase supports vLLM inference, but the speed is slow (20 minutes for a single task). According to my experience, the normal speed is 20 minutes for all tasks.

[Kass123777]

Thank you for your question! This is a known issue. Since the current architecture implements the BaseInference class based on deepspeed and vllm in the same Python file, importing deepspeed-related dependencies causes vllm to fail to start properly. Therefore, I set distributed_executor_backend="ray" when starting vllm. This does significantly affect efficiency. We will further modify the framework in the next version to completely decouple the two backends and fully unleash the inference speed of vllm.