verl icon indicating copy to clipboard operation
verl copied to clipboard
Published 2 weeks ago • verified publisher icon volcengine

vllm开启dp和ep在npu上报错

Open glowwormX opened this issue 2 weeks ago • 0 comments

运行30b 开启：

    actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
    actor_rollout_ref.rollout.data_parallel_size=2 \
    actor_rollout_ref.rollout.expert_parallel_size=4 \

a2上我希望运行235b tp=8 dp=2，或者其他性能更高的切分策略，先在30b上测试结果报错了; 而去除dp ep正常运行，使用fsdp、megatron均报错

verl11.18代码 51d2104ecb61563c41123a8f0bce2f06b18387dc vllm 0.11.0.rc2 cann8.3.rc2

日志：

[36m(WorkerDict pid=3381096)[0m INFO 12-04 17:33:17 [layer.py:332] FlashInfer CUTLASS MoE is currently not available for DP.
[36m(WorkerDict pid=3381096)[0m INFO 12-04 17:33:17 [common_fused_moe.py:239] [EP Rank 3/4] Expert parallelism is enabled. Local/global number of experts: 32/128. Experts local to global index map: 0->96, 1->97, 2->98, 3->99, 4->100, 5->101, 6->102, 7->103, 8->104, 9->105, 10->106, 11->107, 12->108, 13->109, 14->110, 15->111, 16->112, 17->113, 18->114, 19->115, 20->116, 21->117, 22->118, 23->119, 24->120, 25->121, 26->122, 27->123, 28->124, 29->125, 30->126, 31->127.
[36m(WorkerDict pid=3381075)[0m INFO 12-04 17:33:17 [layer.py:1052] [EP Rank 2/4] Expert parallelism is enabled. Expert placement strategy: linear. Local/global number of experts: 32/128. Experts local to global index map: 0->64, 1->65, 2->66, 3->67, 4->68, 5->69, 6->70, 7->71, 8->72, 9->73, 10->74, 11->75, 12->76, 13->77, 14->78, 15->79, 16->80, 17->81, 18->82, 19->83, 20->84, 21->85, 22->86, 23->87, 24->88, 25->89, 26->90, 27->91, 28->92, 29->93, 30->94, 31->95.
[36m(WorkerDict pid=3381075)[0m INFO 12-04 17:33:17 [layer.py:332] FlashInfer CUTLASS MoE is currently not available for DP.
[36m(WorkerDict pid=3381075)[0m INFO 12-04 17:33:17 [common_fused_moe.py:239] [EP Rank 2/4] Expert parallelism is enabled. Local/global number of experts: 32/128. Experts local to global index map: 0->64, 1->65, 2->66, 3->67, 4->68, 5->69, 6->70, 7->71, 8->72, 9->73, 10->74, 11->75, 12->76, 13->77, 14->78, 15->79, 16->80, 17->81, 18->82, 19->83, 20->84, 21->85, 22->86, 23->87, 24->88, 25->89, 26->90, 27->91, 28->92, 29->93, 30->94, 31->95.
[36m(WorkerDict pid=3381089)[0m INFO 12-04 17:33:19 [model_runner_v1.py:2667] Loading model weights took 14.9573 GB
[36m(WorkerDict pid=3381096)[0m INFO 12-04 17:33:19 [model_runner_v1.py:2667] Loading model weights took 14.9573 GB
[36m(WorkerDict pid=3381093)[0m INFO 12-04 17:33:19 [model_runner_v1.py:2667] Loading model weights took 14.9573 GB
[36m(WorkerDict pid=3381094)[0m INFO 12-04 17:33:19 [model_runner_v1.py:2667] Loading model weights took 14.9573 GB
[36m(WorkerDict pid=3381073)[0m INFO 12-04 17:33:19 [model_runner_v1.py:2667] Loading model weights took 14.9573 GB
[36m(WorkerDict pid=3381076)[0m INFO 12-04 17:33:20 [model_runner_v1.py:2667] Loading model weights took 14.9573 GB
[36m(WorkerDict pid=3381074)[0m INFO 12-04 17:33:20 [model_runner_v1.py:2667] Loading model weights took 14.9573 GB
[36m(WorkerDict pid=3381089)[0m WARNING 12-04 17:33:20 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
[36m(WorkerDict pid=3381093)[0m WARNING 12-04 17:33:20 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
[36m(WorkerDict pid=3381094)[0m WARNING 12-04 17:33:20 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
[36m(WorkerDict pid=3381096)[0m WARNING 12-04 17:33:20 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
[36m(WorkerDict pid=3381075)[0m INFO 12-04 17:33:21 [model_runner_v1.py:2667] Loading model weights took 14.9573 GB
[36m(WorkerDict pid=3381074)[0m WARNING 12-04 17:33:21 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
[36m(WorkerDict pid=3381076)[0m WARNING 12-04 17:33:21 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
[36m(WorkerDict pid=3381073)[0m WARNING 12-04 17:33:22 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
[36m(WorkerDict pid=3381075)[0m WARNING 12-04 17:33:22 [cudagraph_dispatcher.py:106] cudagraph dispatching keys are not initialized. No cudagraph will be used.
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275] Error executing method 'determine_available_memory'. This might cause deadlock in distributed execution.
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275] Traceback (most recent call last):
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 267, in execute_method
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return run_method(self, method, args, kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm/utils/__init__.py", line 3122, in run_method
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return func(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm_ascend/worker/worker_v1.py", line 227, in determine_available_memory
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     self.model_runner.profile_run()
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm_ascend/worker/model_runner_v1.py", line 2530, in profile_run
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     hidden_states = self._dummy_run(self.max_num_tokens,
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return func(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm_ascend/worker/model_runner_v1.py", line 2493, in _dummy_run
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     hidden_states = self._generate_dummy_run_hidden_states(
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm_ascend/worker/model_runner_v1.py", line 2314, in _generate_dummy_run_hidden_states
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     hidden_states = self.model(input_ids=input_ids,
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm_ascend/compilation/acl_graph.py", line 113, in __call__
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return self.runnable(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return self._call_impl(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return forward_call(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_moe.py", line 675, in forward
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 310, in __call__
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     output = self.compiled_callable(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/home/ma-user/work/dataset/x30059842_wulan/x30059842/verl/verl_1118/MindSpeed/mindspeed/core/megatron_basic/requirements_basic.py", line 93, in wrapper
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return args[0](*fn_args, **fn_kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_moe.py", line 421, in forward
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     hidden_states, residual = layer(positions, hidden_states, residual)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return self._call_impl(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return forward_call(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_moe.py", line 365, in forward
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     hidden_states = self.mlp(hidden_states)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return self._call_impl(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return forward_call(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_moe.py", line 174, in forward
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     final_hidden_states = self.experts(hidden_states=hidden_states,
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return self._call_impl(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return forward_call(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm/model_executor/custom_op.py", line 44, in forward
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return self._forward_method(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm/model_executor/custom_op.py", line 79, in forward_oot
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return self.forward_native(*args, **kwargs)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/layer.py", line 1827, in forward_native
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     fused_output = torch.ops.vllm.moe_forward(
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/torch/_ops.py", line 1158, in __call__
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return self._op(*args, **(kwargs or {}))
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/layer.py", line 2144, in moe_forward
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     return self.forward_impl(hidden_states, router_logits)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm_ascend/ops/common_fused_moe.py", line 319, in forward_impl
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     hidden_states, router_logits = forward_context.moe_comm_method.prepare(
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm_ascend/ops/moe/moe_comm_method.py", line 73, in prepare
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     hidden_states, router_logits, mc2_mask = self.fused_moe_prepare_finalize.prepare(
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm_ascend/ops/moe/fused_moe_prepare_and_finalize.py", line 489, in prepare
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     hidden_states = self._naive_multicast(hidden_states,
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]   File "/cache/verl_1118_env_main_py310_vllm_v11_rc2_unpack/lib/python3.10/site-packages/vllm_ascend/ops/moe/fused_moe_prepare_and_finalize.py", line 460, in _naive_multicast
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]     buffer[start:end, :].copy_(x)
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275] RuntimeError: copy_d2d_baseformat_opapi:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:88 NPU function error: call aclnnInplaceCopy failed, error code is 161002
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275] [ERROR] 2025-12-04-17:33:31 (PID:3381073, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275] [PID: 3381073] 2025-12-04-17:33:31.901.291 AclNN_Parameter_Error(EZ1001): 4096 and 8192 cannot broadcast.
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]         TraceBack (most recent call last):
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]         The size of tensor self [8192, 2048] must match the size of tensor src [4096, 2048].
[36m(WorkerDict pid=3381073)[0m ERROR 12-04 17:33:31 [worker_base.py:275]

glowwormX avatar Dec 04 '25 10:12 glowwormX