[vllm] - AttributeError: '_OpNamespace' '_vllm_fa2_C' object has no attribute 'varlen_fwd'

[lvhuaizi]

执行： vllm serve /DATA/disk0/ld/ld_model_pretrain/MiniCPM-o-2_6 --dtype auto --max-model-len 2048 --api-key token-abc123 --gpu_memory_utilization 1 --trust-remote-code

报错如下： ERROR 01-26 11:32:37 engine.py:380] File "/autodl-fs/data/github/vllm/vllm/vllm_flash_attn/flash_attn_interface.py", line 154, in flash_attn_varlen_func ERROR 01-26 11:32:37 engine.py:380] out, softmax_lse = torch.ops._vllm_fa2_C.varlen_fwd( AttributeError: '_OpNamespace' '_vllm_fa2_C' object has no attribute 'varlen_fwd'

官网的教程竟然也会报错！ 官网的教程竟然也会报错！ 官网的教程竟然也会报错！

[HwwwwwwwH]

这看起来这报错和我们对 MiniCPM-o-2_6 的前端适配没有关系...可能需要检查一下 CUDA 的版本。 另外我们现在的 MiniCPM-o-2_6 已经合进了vllm官方的仓库中，你可以拿下来再试一下，如果还是有报错，我可以帮你去vllm官方提issue看看。

[biraj-outspeed]

@HwwwwwwwH I'm also facing the same Issue with your fork.

Edit 2: ✅ ✅ ✅ I updated offical vLLM to version 0.7.1 and the above issue was resolved.

I followed the following steps first:

git clone https://github.com/OpenBMB/vllm.git
cd vllm
git checkout minicpmo
python3 -m venv myenv
source myenv/bin/activate
VLLM_USE_PRECOMPILED=1 pip install --editable . --no-cache-dir
export HF_TOKEN=<my-hf-token>
vllm serve openbmb/MiniCPM-o-2_6 \
    --trust-remote-code \
    --max-model-len 2048 \
    --max-num-seq 128

Then I got this error:

  File "/home/ubuntu/biraj/vllm/myenv/lib/python3.12/site-packages/torch/_ops.py", line 1225, in __getattr__
    raise AttributeError(
AttributeError: '_OpNamespace' '_vllm_fa2_C' object has no attribute 'varlen_fwd'

Here's is the full error trace.

I also tried using official vLLM since I noticed that openbmb/MiniCPM-o-2_6 is listed in vLLM's supported models. However, running vLLM's docker image raised error saying that the model is not supported.

Command:

    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN=<my-hf-token>" \
    -p 8000:8000 \
    --ipc=host \
    -d \
    vllm/vllm-openai:latest \
    --model openbmb/MiniCPM-o-2_6 \
    --trust-remote-code \
    --max-model-len 2048 \
    --max-num-seq 128

Error:

ValueError: Model architectures ['MiniCPMO'] are not supported for now.

Edit 1: I get the same error for vLLM 0.7.0 without Docker.

~/biraj/minicpmo$ vllm --version
INFO 02-03 04:52:09 __init__.py:183] Automatically detected platform cuda.
0.7.0

Command: Same vllm serve that I mentioned at the begging of this comment.

Error:

ValueError: Model architectures ['MiniCPMO'] are not supported for now.

Edit 2: I updated vLLM to version 0.7.1 and the above issue was resolved.

[YiJunShenS]

这看起来这报错和我们对 MiniCPM-o-2_6 的前端适配没有关系...可能需要检查一下 CUDA 的版本。 另外我们现在的 MiniCPM-o-2_6 已经合进了vllm官方的仓库中，你可以拿下来再试一下，如果还是有报错，我可以帮你去vllm官方提issue看看。 你好，我使用了最新版本的vllm==0.7.1，然后运行时候报了另一个错：AttributeError: 'MiniCPMOProcessor' object has no attribute 'get_audio_placeholder'

[HwwwwwwwH]

这看起来这报错和我们对 MiniCPM-o-2_6 的前端适配没有关系...可能需要检查一下 CUDA 的版本。 另外我们现在的 MiniCPM-o-2_6 已经合进了vllm官方的仓库中，你可以拿下来再试一下，如果还是有报错，我可以帮你去vllm官方提issue看看。 你好，我使用了最新版本的vllm==0.7.1，然后运行时候报了另一个错：AttributeError: 'MiniCPMOProcessor' object has no attribute 'get_audio_placeholder'

还有更多报错的 traceback 吗？

[YiJunShenS]

这看起来这报错和我们对 MiniCPM-o-2_6 的前端适配没有关系...可能需要检查一下 CUDA 的版本。 另外我们现在的 MiniCPM-o-2_6 已经合进了vllm官方的仓库中，你可以拿下来再试一下，如果还是有报错，我可以帮你去vllm官方提issue看看。 你好，我使用了最新版本的vllm==0.7.1，然后运行时候报了另一个错：AttributeError: 'MiniCPMOProcessor' object has no attribute 'get_audio_placeholder'

还有更多报错的 traceback 吗？

您好，以下是具体的报错的traceback： [rank0]: Traceback (most recent call last): [rank0]: File "/U03/syj/test.py", line 10, in [rank0]: llm = LLM( [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/utils.py", line 1039, in inner [rank0]: return fn(*args, **kwargs) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 240, in init [rank0]: self.llm_engine = self.engine_class.from_engine_args( [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 482, in from_engine_args [rank0]: engine = cls( [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 274, in init [rank0]: self._initialize_kv_caches() [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 414, in _initialize_kv_caches [rank0]: self.model_executor.determine_num_available_blocks()) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 99, in determine_num_available_blocks [rank0]: results = self.collective_rpc("determine_num_available_blocks") [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 49, in collective_rpc [rank0]: answer = run_method(self.driver_worker, method, args, kwargs) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/utils.py", line 2208, in run_method [rank0]: return func(*args, **kwargs) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context [rank0]: return func(*args, **kwargs) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/worker/worker.py", line 228, in determine_num_available_blocks [rank0]: self.model_runner.profile_run() [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context [rank0]: return func(*args, **kwargs) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1236, in profile_run [rank0]: self._dummy_run(max_num_batched_tokens, max_num_seqs) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1301, in _dummy_run [rank0]: .dummy_data_for_profiling(self.model_config, [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/inputs/registry.py", line 333, in dummy_data_for_profiling [rank0]: dummy_data = profiler.get_dummy_data(seq_len) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/multimodal/profiling.py", line 161, in get_dummy_data [rank0]: mm_inputs = self._get_dummy_mm_inputs(seq_len, mm_counts) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/multimodal/profiling.py", line 139, in _get_dummy_mm_inputs [rank0]: return self.processor.apply( [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/model_executor/models/minicpmv.py", line 803, in apply [rank0]: result = super().apply(prompt, mm_data, hf_processor_mm_kwargs) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 1230, in apply [rank0]: hf_mm_placeholders = self._find_mm_placeholders( [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 793, in _find_mm_placeholders [rank0]: return find_mm_placeholders(mm_prompt_repls, new_token_ids, [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 579, in find_mm_placeholders [rank0]: return dict(full_groupby_modality(it)) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 184, in full_groupby_modality [rank0]: return full_groupby(values, key=lambda x: x.modality) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/utils.py", line 873, in full_groupby [rank0]: for value in values: [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 534, in _iter_placeholders [rank0]: replacement = repl_info.get_replacement(item_idx) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 270, in get_replacement [rank0]: replacement = replacement(item_idx) [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/model_executor/models/minicpmo.py", line 355, in get_replacement_minicpmv [rank0]: return self.get_audio_prompt_texts( [rank0]: File "/root/miniconda3/envs/minicpm-o/lib/python3.10/site-packages/vllm/model_executor/models/minicpmo.py", line 232, in get_audio_prompt_texts [rank0]: return self.info.get_hf_processor().get_audio_placeholder( [rank0]: AttributeError: 'MiniCPMOProcessor' object has no attribute 'get_audio_placeholder'

[HwwwwwwwH]

可以更新一下最新HF仓库的代码。

[YiJunShenS]

可以更新一下最新HF仓库的代码。

你好，解决了谢谢~