Qwen3 icon indicating copy to clipboard operation
Qwen3 copied to clipboard
verified publisher icon QwenLM

[Bug]: vllm serve /data/Qwen3-30B-A3B-GPTQ-Int4/ 4090 24G显卡启动不起来

Open zhaoliubox opened this issue 11 months ago • 1 comments

Model Series

Qwen3

What are the models used?

Qwen3-30B-A3B-GPTQ-Int4

What is the scenario where the problem happened?

[进行部署][vllm]

Is this a known issue?

  • [x] I have followed the GitHub README.
  • [x] I have checked the Qwen documentation and cannot find an answer there.
  • [x] I have checked the documentation of the related framework and cannot find useful information.
  • [x] I have searched the issues and there is not a similar one.

Information about environment

操作系统：Ubuntu 22.04.4 LTS Python：Python 3.10.12 GPU: 4090 NVIDIA 驱动程序:560.35.03 CUDA 编译器：12.6.r12.6 PyTorch：2.6.0+cu124

Log output

ERROR 05-23 09:14:03 [core.py:396] EngineCore failed to start.
ERROR 05-23 09:14:03 [core.py:396] Traceback (most recent call last):
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
ERROR 05-23 09:14:03 [core.py:396]     self.dispatch_table[inst.opcode](self, inst)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 659, in wrapper
ERROR 05-23 09:14:03 [core.py:396]     return inner_fn(self, inst)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1658, in CALL_FUNCTION
ERROR 05-23 09:14:03 [core.py:396]     self.call_function(fn, args, {})
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 897, in call_function
ERROR 05-23 09:14:03 [core.py:396]     self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 914, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return variables.UserFunctionVariable(fn, source=source).call_function(
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 317, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return super().call_function(tx, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 118, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 903, in inline_user_function_return
ERROR 05-23 09:14:03 [core.py:396]     return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 234, in patched_inline_call
ERROR 05-23 09:14:03 [core.py:396]     return inline_call(parent, func, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3072, in inline_call
ERROR 05-23 09:14:03 [core.py:396]     return cls.inline_call_(parent, func, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3198, in inline_call_
ERROR 05-23 09:14:03 [core.py:396]     tracer.run()
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
ERROR 05-23 09:14:03 [core.py:396]     while self.step():
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 967, in step
ERROR 05-23 09:14:03 [core.py:396]     self.exception_handler(e)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1552, in exception_handler
ERROR 05-23 09:14:03 [core.py:396]     raise raised_exception
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
ERROR 05-23 09:14:03 [core.py:396]     self.dispatch_table[inst.opcode](self, inst)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 659, in wrapper
ERROR 05-23 09:14:03 [core.py:396]     return inner_fn(self, inst)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1658, in CALL_FUNCTION
ERROR 05-23 09:14:03 [core.py:396]     self.call_function(fn, args, {})
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 897, in call_function
ERROR 05-23 09:14:03 [core.py:396]     self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 170, in realize_and_forward
ERROR 05-23 09:14:03 [core.py:396]     return getattr(self.realize(), name)(*args, **kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 914, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return variables.UserFunctionVariable(fn, source=source).call_function(
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 317, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return super().call_function(tx, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 118, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 903, in inline_user_function_return
ERROR 05-23 09:14:03 [core.py:396]     return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 234, in patched_inline_call
ERROR 05-23 09:14:03 [core.py:396]     return inline_call(parent, func, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3072, in inline_call
ERROR 05-23 09:14:03 [core.py:396]     return cls.inline_call_(parent, func, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3198, in inline_call_
ERROR 05-23 09:14:03 [core.py:396]     tracer.run()
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
ERROR 05-23 09:14:03 [core.py:396]     while self.step():
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 967, in step
ERROR 05-23 09:14:03 [core.py:396]     self.exception_handler(e)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1552, in exception_handler
ERROR 05-23 09:14:03 [core.py:396]     raise raised_exception
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
ERROR 05-23 09:14:03 [core.py:396]     self.dispatch_table[inst.opcode](self, inst)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 659, in wrapper
ERROR 05-23 09:14:03 [core.py:396]     return inner_fn(self, inst)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1748, in CALL_FUNCTION_KW
ERROR 05-23 09:14:03 [core.py:396]     self.call_function(fn, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 897, in call_function
ERROR 05-23 09:14:03 [core.py:396]     self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 170, in realize_and_forward
ERROR 05-23 09:14:03 [core.py:396]     return getattr(self.realize(), name)(*args, **kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 914, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return variables.UserFunctionVariable(fn, source=source).call_function(
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 317, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return super().call_function(tx, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 118, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 903, in inline_user_function_return
ERROR 05-23 09:14:03 [core.py:396]     return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 234, in patched_inline_call
ERROR 05-23 09:14:03 [core.py:396]     return inline_call(parent, func, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3072, in inline_call
ERROR 05-23 09:14:03 [core.py:396]     return cls.inline_call_(parent, func, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3198, in inline_call_
ERROR 05-23 09:14:03 [core.py:396]     tracer.run()
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
ERROR 05-23 09:14:03 [core.py:396]     while self.step():
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 967, in step
ERROR 05-23 09:14:03 [core.py:396]     self.exception_handler(e)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1552, in exception_handler
ERROR 05-23 09:14:03 [core.py:396]     raise raised_exception
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
ERROR 05-23 09:14:03 [core.py:396]     self.dispatch_table[inst.opcode](self, inst)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 659, in wrapper
ERROR 05-23 09:14:03 [core.py:396]     return inner_fn(self, inst)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1658, in CALL_FUNCTION
ERROR 05-23 09:14:03 [core.py:396]     self.call_function(fn, args, {})
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 897, in call_function
ERROR 05-23 09:14:03 [core.py:396]     self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 378, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return super().call_function(tx, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 317, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return super().call_function(tx, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 118, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 903, in inline_user_function_return
ERROR 05-23 09:14:03 [core.py:396]     return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 234, in patched_inline_call
ERROR 05-23 09:14:03 [core.py:396]     return inline_call(parent, func, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3072, in inline_call
ERROR 05-23 09:14:03 [core.py:396]     return cls.inline_call_(parent, func, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3198, in inline_call_
ERROR 05-23 09:14:03 [core.py:396]     tracer.run()
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
ERROR 05-23 09:14:03 [core.py:396]     while self.step():
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 967, in step
ERROR 05-23 09:14:03 [core.py:396]     self.exception_handler(e)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1552, in exception_handler
ERROR 05-23 09:14:03 [core.py:396]     raise raised_exception
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
ERROR 05-23 09:14:03 [core.py:396]     self.dispatch_table[inst.opcode](self, inst)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 659, in wrapper
ERROR 05-23 09:14:03 [core.py:396]     return inner_fn(self, inst)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1748, in CALL_FUNCTION_KW
ERROR 05-23 09:14:03 [core.py:396]     self.call_function(fn, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 897, in call_function
ERROR 05-23 09:14:03 [core.py:396]     self.push(fn.call_function(self, args, kwargs))  # type: ignore[arg-type]
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 378, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return super().call_function(tx, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 317, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return super().call_function(tx, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 118, in call_function
ERROR 05-23 09:14:03 [core.py:396]     return tx.inline_user_function_return(self, [*self.self_args(), *args], kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 903, in inline_user_function_return
ERROR 05-23 09:14:03 [core.py:396]     return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 234, in patched_inline_call
ERROR 05-23 09:14:03 [core.py:396]     return inline_call(parent, func, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3072, in inline_call
ERROR 05-23 09:14:03 [core.py:396]     return cls.inline_call_(parent, func, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 3198, in inline_call_
ERROR 05-23 09:14:03 [core.py:396]     tracer.run()
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
ERROR 05-23 09:14:03 [core.py:396]     while self.step():
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 967, in step
ERROR 05-23 09:14:03 [core.py:396]     self.exception_handler(e)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1552, in exception_handler
ERROR 05-23 09:14:03 [core.py:396]     raise raised_exception
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
ERROR 05-23 09:14:03 [core.py:396]     self.dispatch_table[inst.opcode](self, inst)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1444, in RAISE_VARARGS
ERROR 05-23 09:14:03 [core.py:396]     self._raise_exception_variable(inst)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1437, in _raise_exception_variable
ERROR 05-23 09:14:03 [core.py:396]     raise exc.ObservedException(f"raised exception {val}")
ERROR 05-23 09:14:03 [core.py:396] torch._dynamo.exc.ObservedException: raised exception ExceptionVariable()
ERROR 05-23 09:14:03 [core.py:396]
ERROR 05-23 09:14:03 [core.py:396] During handling of the above exception, another exception occurred:
ERROR 05-23 09:14:03 [core.py:396]
ERROR 05-23 09:14:03 [core.py:396] Traceback (most recent call last):
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 387, in run_engine_core
ERROR 05-23 09:14:03 [core.py:396]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 329, in __init__
ERROR 05-23 09:14:03 [core.py:396]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 71, in __init__
ERROR 05-23 09:14:03 [core.py:396]     self._initialize_kv_caches(vllm_config)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 129, in _initialize_kv_caches
ERROR 05-23 09:14:03 [core.py:396]     available_gpu_memory = self.model_executor.determine_available_memory()
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
ERROR 05-23 09:14:03 [core.py:396]     output = self.collective_rpc("determine_available_memory")
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 05-23 09:14:03 [core.py:396]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/utils.py", line 2456, in run_method
ERROR 05-23 09:14:03 [core.py:396]     return func(*args, **kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 05-23 09:14:03 [core.py:396]     return func(*args, **kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 183, in determine_available_memory
ERROR 05-23 09:14:03 [core.py:396]     self.model_runner.profile_run()
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1651, in profile_run
ERROR 05-23 09:14:03 [core.py:396]     hidden_states = self._dummy_run(self.max_num_tokens)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 05-23 09:14:03 [core.py:396]     return func(*args, **kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1497, in _dummy_run
ERROR 05-23 09:14:03 [core.py:396]     outputs = model(
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
ERROR 05-23 09:14:03 [core.py:396]     return self._call_impl(*args, **kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
ERROR 05-23 09:14:03 [core.py:396]     return forward_call(*args, **kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_moe.py", line 509, in forward
ERROR 05-23 09:14:03 [core.py:396]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 238, in __call__
ERROR 05-23 09:14:03 [core.py:396]     output = self.compiled_callable(*args, **kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 574, in _fn
ERROR 05-23 09:14:03 [core.py:396]     return fn(*args, **kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1380, in __call__
ERROR 05-23 09:14:03 [core.py:396]     return self._torchdynamo_orig_callable(
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 547, in __call__
ERROR 05-23 09:14:03 [core.py:396]     return _compile(
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 986, in _compile
ERROR 05-23 09:14:03 [core.py:396]     guarded_code = compile_inner(code, one_graph, hooks, transform)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 715, in compile_inner
ERROR 05-23 09:14:03 [core.py:396]     return _compile_inner(code, one_graph, hooks, transform)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_utils_internal.py", line 95, in wrapper_function
ERROR 05-23 09:14:03 [core.py:396]     return function(*args, **kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 750, in _compile_inner
ERROR 05-23 09:14:03 [core.py:396]     out_code = transform_code_object(code, transform)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1361, in transform_code_object
ERROR 05-23 09:14:03 [core.py:396]     transformations(instructions, code_options)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 231, in _fn
ERROR 05-23 09:14:03 [core.py:396]     return fn(*args, **kwargs)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 662, in transform
ERROR 05-23 09:14:03 [core.py:396]     tracer.run()
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2868, in run
ERROR 05-23 09:14:03 [core.py:396]     super().run()
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
ERROR 05-23 09:14:03 [core.py:396]     while self.step():
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 967, in step
ERROR 05-23 09:14:03 [core.py:396]     self.exception_handler(e)
ERROR 05-23 09:14:03 [core.py:396]   File "/home/oem/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1551, in exception_handler
ERROR 05-23 09:14:03 [core.py:396]     raise Unsupported("Observed exception")
ERROR 05-23 09:14:03 [core.py:396] torch._dynamo.exc.Unsupported: Observed exception
ERROR 05-23 09:14:03 [core.py:396]
ERROR 05-23 09:14:03 [core.py:396] from user code:
ERROR 05-23 09:14:03 [core.py:396]    File "/home/oem/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_moe.py", line 369, in forward
ERROR 05-23 09:14:03 [core.py:396]     hidden_states, residual = layer(positions, hidden_states, residual)
ERROR 05-23 09:14:03 [core.py:396]
ERROR 05-23 09:14:03 [core.py:396] Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
ERROR 05-23 09:14:03 [core.py:396]
ERROR 05-23 09:14:03 [core.py:396]
ERROR 05-23 09:14:03 [core.py:396] You can suppress this exception and fall back to eager by setting:
ERROR 05-23 09:14:03 [core.py:396]     import torch._dynamo
ERROR 05-23 09:14:03 [core.py:396]     torch._dynamo.config.suppress_errors = True
ERROR 05-23 09:14:03 [core.py:396]

Description

我使用了vllm serve --gpu_memory_utilization=0.93 --max_model_len=11000 --max-num-seqs=1 --enable-auto-tool-choice --tool-call-parser=hermes /data/Qwen3-32B-AWQ/ 运行Qwen3-32B-AWQ是能正常运行的， 但是运行Qwen3-30B-A3B-GPTQ-Int4运行不起来， 换了最简洁的命令也是运行不起来vllm serve /data/Qwen3-30B-A3B-GPTQ-Int4/

zhaoliubox avatar May 23 '25 01:05 zhaoliubox

同样的问题，请问解决了吗

KeDaCoYa avatar May 26 '25 09:05 KeDaCoYa

解决了吗？我vllm serve启动会报这个错误 assert quant_method is not None

JohnLoveMm avatar Jun 04 '25 06:06 JohnLoveMm

GPTQ for Qwen3-MoE isn't supported by vllm until 0.9.0. In addition, you may need more than a single GPU to serve Qwen3-30B-A3B-GPTQ-Int4 if sufficient context length is needed.

jklj077 avatar Jun 04 '25 07:06 jklj077

This issue has been automatically marked as inactive due to lack of recent activity. Should you believe it remains unresolved and warrants attention, kindly leave a comment on this thread.

github-actions[bot] avatar Jul 04 '25 08:07 github-actions[bot]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

github-actions[bot] avatar Aug 11 '25 08:08 github-actions[bot]