vllm
vllm copied to clipboard
vllm-project
Support Deepseek-V2
Description:
This PR introduces support for the recently released DeepSeek-V2 model by DeepSeek-AI.
Key Updates:
- Model Integration: Successfully integrated the DeepSeek-V2 model, developed by the DeepSeek-AI team, aiming to provide advanced natural language processing capabilities.
Related Resources:
- Model Repository: DeepSeek-V2 Model Repository
- Technical Report: DeepSeek-V2 Technical Report
Todo:
- Efficient Inference Mode: Implement the efficient inference mode described in the paper.
We look forward to community feedback and suggestions to help us continuously improve and refine the integration and inference implementation of the DeepSeek-V2 model.
Testing
from vllm import LLM, SamplingParams
# Sample prompts.
prompts = [
"User: The future of AI is? Assistant:"
]
# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.0, top_p=1, max_tokens=32)
# Create an LLM.
llm = LLM(model="deepseek-ai/DeepSeek-V2-Chat", tensor_parallel_size=8, max_num_seqs = 1, max_model_len = 1024, trust_remote_code=True, enforce_eager = True)
# Generate texts from the prompts. The output is a list of RequestOutput objects
# that contain the prompt, generated text, and other information.
outputs = llm.generate(prompts, sampling_params)
# Print the outputs.
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
Prompt: 'User: The future of AI is? Assistant:', Generated text: ' The future of AI, or Artificial Intelligence, is a topic of much speculation and debate. AI has the potential to revolutionize many aspects of our lives, from'
Note: Currently, only the inference method using the Multi-Head Attention (MHA) approach has been implemented, and the efficient inference mode mentioned in the paper has not yet been realized.
ERROR 05-08 20:22:08 worker_base.py:145] ValueError: Model architectures ['DeepseekV2ForCausalLM'] are not supported for now. Supported architectures: ['AquilaModel', 'AquilaForCausalLM', 'BaiChuanForCausalLM', 'BaichuanForCausalLM', 'BloomForCausalLM', 'ChatGLMModel', 'ChatGLMForConditionalGeneration', 'CohereForCausalLM', 'DbrxForCausalLM', 'DeciLMForCausalLM', 'DeepseekForCausalLM', 'FalconForCausalLM', 'GemmaForCausalLM', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTJForCausalLM', 'GPTNeoXForCausalLM', 'InternLMForCausalLM', 'InternLM2ForCausalLM', 'JAISLMHeadModel', 'LlamaForCausalLM', 'LlavaForConditionalGeneration', 'LLaMAForCausalLM', 'MistralForCausalLM', 'MixtralForCausalLM', 'QuantMixtralForCausalLM', 'MptForCausalLM', 'MPTForCausalLM', 'MiniCPMForCausalLM', 'OlmoForCausalLM', 'OPTForCausalLM', 'OrionForCausalLM', 'PhiForCausalLM', 'Phi3ForCausalLM', 'QWenLMHeadModel', 'Qwen2ForCausalLM', 'Qwen2MoeForCausalLM', 'RWForCausalLM', 'StableLMEpochForCausalLM', 'StableLmForCausalLM', 'Starcoder2ForCausalLM', 'XverseForCausalLM']
Currently, only the inference method using the Multi-Head Attention (MHA) approach has been implemented, and the efficient inference mode mentioned in the paper has not yet been realized.
What's the reason it is not supported in this PR?
Hi, with only MHA, is it possible to realize max_model_len = 128k? In my test, may only 12k.
What's the reason it is not supported in this PR?
The internal inference implementation supports MLA. The implementation on vLLM is more about making it support quickly and matching the model parameters with the code. So the efficiency of using it for LLM Serving is not high enough. I think maybe the current PR could be quickly reviewed and merged asap. Subsequent communities can consider implementing an integrated version.
Hi @zwd003 May you merge the latest main branch and fix the conflicts? Thanks.
Hi @zwd003 May you merge the latest main branch and fix the conflicts? Thanks.
ok
HI @zwd003 This error occurred during the deployment process. How to solve it? Thanks!
(RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] File "/opt/vllm/vllm/model_executor/models/deepseek_v2.py", line 156, in forward (RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] final_hidden_states = fused_moe(hidden_states, (RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] TypeError: fused_moe() got an unexpected keyword argument 'num_expert_group'
HI @zwd003 This error occurred during the deployment process. How to solve it? Thanks!
(RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] File "/opt/vllm/vllm/model_executor/models/deepseek_v2.py", line 156, in forward (RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] final_hidden_states = fused_moe(hidden_states, (RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] TypeError: fused_moe() got an unexpected keyword argument 'num_expert_group'
I encountered the same error
HI @zwd003 This error occurred during the deployment process. How to solve it? Thanks!
(RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] File "/opt/vllm/vllm/model_executor/models/deepseek_v2.py", line 156, in forward (RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] final_hidden_states = fused_moe(hidden_states, (RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] TypeError: fused_moe() got an unexpected keyword argument 'num_expert_group'
git checkout 5688e58ca2797a34bd56e75c045d41be6aca1e2b solved this problem
HI @zwd003 This error occurred during the deployment process. How to solve it? Thanks! (RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] File "/opt/vllm/vllm/model_executor/models/deepseek_v2.py", line 156, in forward (RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] final_hidden_states = fused_moe(hidden_states, (RayWorkerWrapper pid=52311) ERROR 05-11 18:04:33 worker_base.py:145] TypeError: fused_moe() got an unexpected keyword argument 'num_expert_group'
git checkout 5688e58ca2797a34bd56e75c045d41be6aca1e2bsolved this problem
Thanks! :D
Hi @zwd003 May you merge the latest main branch and fix the conflicts? Thanks.
ok
hello,I encountered this error when the QPS was increased to 2.
[' 根据指令"周日晚上"，我们将按照步骤进行处理：\n\n1. 选择']
INFO:werkzeug:172.16.178.41 - - [13/May/2024 12:31:52] "POST /get_data HTTP/1.1" 200 -
Processed prompts: 0%| | 0/1 [00:00<?, ?it/s](RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] Error executing method execute_model. This might cause deadlock in distributed execution. | 0/2 [00:00<?, ?it/s]
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] Traceback (most recent call last):
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/workspace/[email protected]/code/vllm/vllm/worker/worker_base.py", line 137, in execute_method
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] return executor(*args, **kwargs)
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] return func(*args, **kwargs)
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/workspace/[email protected]/code/vllm/vllm/worker/worker.py", line 249, in execute_model
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] output = self.model_runner.execute_model(seq_group_metadata_list,
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] return func(*args, **kwargs)
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/workspace/[email protected]/code/vllm/vllm/worker/model_runner.py", line 787, in execute_model
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] ) = self.prepare_input_tensors(seq_group_metadata_list)
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/workspace/[email protected]/code/vllm/vllm/worker/model_runner.py", line 729, in prepare_input_tensors
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] input_tokens = metadata_dict.pop("input_tokens")
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] KeyError: 'input_tokens'
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] Error executing method execute_model. This might cause deadlock in distributed execution.
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] Traceback (most recent call last):
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/workspace/[email protected]/code/vllm/vllm/worker/worker_base.py", line 137, in execute_method
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] return executor(*args, **kwargs)
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] return func(*args, **kwargs)
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/workspace/[email protected]/code/vllm/vllm/worker/worker.py", line 237, in execute_model
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] data = broadcast_tensor_dict(src=0)
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/workspace/[email protected]/code/vllm/vllm/distributed/communication_op.py", line 216, in broadcast_tensor_dict
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] torch.distributed.broadcast_object_list(recv_metadata_list,
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] return func(*args, **kwargs)
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2674, in broadcast_object_list
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] object_list[i] = _tensor_to_object(obj_view, obj_size, group)
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2362, in _tensor_to_object
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] return _unpickler(io.BytesIO(buf)).load()
(RayWorkerWrapper pid=1539303) ERROR 05-13 12:31:53 worker_base.py:145] _pickle.UnpicklingError: invalid load key, '\xea'.
(RayWorkerWrapper pid=1542773) INFO 05-13 12:26:25 model_runner.py:175] Loading model weights took 56.1087 GB [repeated 6x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO Connected all trees [repeated 7x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 [repeated 7x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO Using non-device net plugin version 0 [repeated 7x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO comm 0x55f8f5a608b0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId b3000 commId 0x7b5f29ff7a9fb9f5 - Init START [repeated 7x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO NVLS multicast support is not available on dev 7 [repeated 7x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO comm 0x55f8f5a608b0 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0 [repeated 7x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO 16 coll channels, 0 collnet channels, 0 nvls channels, 16 p2p channels, 16 p2p channels per peer [repeated 7x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO comm 0x55f8f5a608b0 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId b3000 commId 0x7b5f29ff7a9fb9f5 - Init COMPLETE [repeated 7x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2076947 [7] NCCL INFO Channel 15/1 : 7[7] -> 0[0] via P2P/CUMEM/read [repeated 336x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO Connected all rings [repeated 7x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO Using network IB [repeated 6x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO bootstrapSplit: comm 0x55f8f5a608b0 parent 0x55f8e5006f90 rank 7 nranks 8 color -934961569 key 7 prev 6 next 0 - DONE [repeated 6x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO Setting affinity for GPU 7 to ffffffff,00000000,ffffffff,00000000 [repeated 6x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [repeated 6x across cluster]
(RayWorkerWrapper pid=1542773) cnwla-a800-p01009:1542773:2075575 [7] NCCL INFO P2P Chunksize set to 524288 [repeated 6x across cluster]
加载模型时报如下错误：
Cache shape torch.Size([163840, 64]) [repeated 6x across cluster]
INFO 05-14 22:41:26 model_runner.py:166] Loading model weights took 56.1087 GB
/tmp/tmpw9q1ie7x/main.c: In function ‘list_to_cuuint64_array’:
/tmp/tmpw9q1ie7x/main.c:354:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
for (Py_ssize_t i = 0; i < len; i++) {
^
/tmp/tmpw9q1ie7x/main.c:354:3: note: use option -std=c99 or -std=gnu99 to compile your code
/tmp/tmpw9q1ie7x/main.c: In function ‘list_to_cuuint32_array’:
/tmp/tmpw9q1ie7x/main.c:365:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
for (Py_ssize_t i = 0; i < len; i++) {
^
ERROR 05-14 22:41:31 worker_base.py:145] Error executing method determine_num_available_blocks. This might cause deadlock in distributed execution.
ERROR 05-14 22:41:31 worker_base.py:145] Traceback (most recent call last):
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/dolphinfs_hdd_hadoop-mtai/users/fengxin09/vllm_n/vllm/vllm/worker/worker_base.py", line 137, in execute_method
ERROR 05-14 22:41:31 worker_base.py:145] return executor(*args, **kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/.local/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR 05-14 22:41:31 worker_base.py:145] return func(*args, **kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/dolphinfs_hdd_hadoop-mtai/users/fengxin09/vllm_n/vllm/vllm/worker/worker.py", line 141, in determine_num_available_blocks
ERROR 05-14 22:41:31 worker_base.py:145] self.model_runner.profile_run()
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/.local/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR 05-14 22:41:31 worker_base.py:145] return func(*args, **kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/dolphinfs_hdd_hadoop-mtai/users/fengxin09/vllm_n/vllm/vllm/worker/model_runner.py", line 873, in profile_run
ERROR 05-14 22:41:31 worker_base.py:145] self.execute_model(seqs, kv_caches)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/.local/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR 05-14 22:41:31 worker_base.py:145] return func(*args, **kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/dolphinfs_hdd_hadoop-mtai/users/fengxin09/vllm_n/vllm/vllm/worker/model_runner.py", line 792, in execute_model
ERROR 05-14 22:41:31 worker_base.py:145] hidden_states = model_executable(**execute_model_kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
ERROR 05-14 22:41:31 worker_base.py:145] return self._call_impl(*args, **kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
ERROR 05-14 22:41:31 worker_base.py:145] return forward_call(*args, **kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/dolphinfs_hdd_hadoop-mtai/users/fengxin09/vllm_n/vllm/vllm/model_executor/models/deepseek_v2.py", line 429, in forward
ERROR 05-14 22:41:31 worker_base.py:145] hidden_states = self.model(input_ids, positions, kv_caches,
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
ERROR 05-14 22:41:31 worker_base.py:145] return self._call_impl(*args, **kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
ERROR 05-14 22:41:31 worker_base.py:145] return forward_call(*args, **kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/dolphinfs_hdd_hadoop-mtai/users/fengxin09/vllm_n/vllm/vllm/model_executor/models/deepseek_v2.py", line 400, in forward
ERROR 05-14 22:41:31 worker_base.py:145] hidden_states, residual = layer(positions, hidden_states,
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
ERROR 05-14 22:41:31 worker_base.py:145] return self._call_impl(*args, **kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
ERROR 05-14 22:41:31 worker_base.py:145] return forward_call(*args, **kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/dolphinfs_hdd_hadoop-mtai/users/fengxin09/vllm_n/vllm/vllm/model_executor/models/deepseek_v2.py", line 362, in forward
ERROR 05-14 22:41:31 worker_base.py:145] hidden_states = self.mlp(hidden_states)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
ERROR 05-14 22:41:31 worker_base.py:145] return self._call_impl(*args, **kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
ERROR 05-14 22:41:31 worker_base.py:145] return forward_call(*args, **kwargs)
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/dolphinfs_hdd_hadoop-mtai/users/fengxin09/vllm_n/vllm/vllm/model_executor/models/deepseek_v2.py", line 156, in forward
ERROR 05-14 22:41:31 worker_base.py:145] final_hidden_states = fused_moe(hidden_states,
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/dolphinfs_hdd_hadoop-mtai/users/fengxin09/vllm_n/vllm/vllm/model_executor/layers/fused_moe/fused_moe.py", line 529, in fused_moe
ERROR 05-14 22:41:31 worker_base.py:145] return fused_experts(hidden_states,
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/dolphinfs_hdd_hadoop-mtai/users/fengxin09/vllm_n/vllm/vllm/model_executor/layers/fused_moe/fused_moe.py", line 439, in fused_experts
ERROR 05-14 22:41:31 worker_base.py:145] invoke_fused_moe_kernel(hidden_states,
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/dolphinfs_hdd_hadoop-mtai/users/fengxin09/vllm_n/vllm/vllm/model_executor/layers/fused_moe/fused_moe.py", line 246, in invoke_fused_moe_kernel
ERROR 05-14 22:41:31 worker_base.py:145] fused_moe_kernel[grid](
ERROR 05-14 22:41:31 worker_base.py:145] File "/home/hadoop-mtai/.local/lib/python3.9/site-packages/triton/runtime/jit.py", line 167, in
Process finished with exit code 1
BTW, I found there were bugs when setting the model type as torch.half.
With your official vllm example (bf16), the generated results are correct:
However, set dtype=torch.half while creating the LLM object, the results are semanticless：
@zwd003 I did the refactoring of the MoE code for you, can you look into the other comments I just added?
I used gptq(int4) method to quantize deepseek_v2 model. When i load quantized model with vLLM，i got below error:
vLLM parameters:
--dtype float16 --load-format safetensors --trust-remote-code --tensor-parallel-size 2 --enforce-eager --device cuda --max-model-len 1024
generated config.json
"quantization_config": {
"bits": 4,
"damp_percent": 0.1,
"desc_act": false,
"group_size": 128,
"modules_in_block_to_quantize": null,
"quant_method": "gptq",
"sym": true,
"true_sequential": true
}
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] Error executing method determine_num_available_blocks. This might cause deadlock in distributed execution.
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] Traceback (most recent call last):
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 137, in execute_method
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] return executor(*args, **kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] return func(*args, **kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 139, in determine_num_available_blocks
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] self.model_runner.profile_run()
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] return func(*args, **kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 888, in profile_run
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] self.execute_model(seqs, kv_caches)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] return func(*args, **kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 808, in execute_model
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] hidden_states = model_executable(**execute_model_kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] return self._call_impl(*args, **kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] return forward_call(*args, **kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/deepseek_v2.py", line 429, in forward
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] hidden_states = self.model(input_ids, positions, kv_caches,
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] return self._call_impl(*args, **kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] return forward_call(*args, **kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/deepseek_v2.py", line 400, in forward
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] hidden_states, residual = layer(positions, hidden_states,
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] return self._call_impl(*args, **kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] return forward_call(*args, **kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/deepseek_v2.py", line 362, in forward
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] hidden_states = self.mlp(hidden_states)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] return self._call_impl(*args, **kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] return forward_call(*args, **kwargs)
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/deepseek_v2.py", line 156, in forward
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] final_hidden_states = fused_moe(hidden_states,
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/fused_moe/fused_moe.py", line 357, in fused_moe
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] assert hidden_states.shape[1] == w1.shape[2], "Hidden size mismatch"
(RayWorkerWrapper pid=9166) ERROR 05-20 07:10:58 worker_base.py:145] AssertionError: Hidden size mismatch
I follow the file change to change the file ,but when I use the below code :
python -m vllm.entrypoints.openai.api_server --model /root/DeepSeek-V2-Chat --trust-remote-code
there is a error happen:
INFO 05-22 06:30:51 llm_engine.py:103] Initializing an LLM engine (v0.4.2) with config: model='/root/DeepSeek-V2-Chat', speculative_config=None, tokenizer='/root/DeepSeek-V2-Chat', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=163840, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/root/DeepSeek-V2-Chat)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 05-22 06:30:52 selector.py:44] Using FlashAttention-2 backend.
[rank0]: Traceback (most recent call last):
[rank0]: File "/opt/conda/envs/deepseek/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank0]: return _run_code(code, main_globals, None,
[rank0]: File "/opt/conda/envs/deepseek/lib/python3.10/runpy.py", line 86, in _run_code
[rank0]: exec(code, run_globals)
[rank0]: File "/root/vllm/vllm/entrypoints/openai/api_server.py", line 186, in <module>
[rank0]: engine = AsyncLLMEngine.from_engine_args(
[rank0]: File "/root/vllm/vllm/engine/async_llm_engine.py", line 374, in from_engine_args
[rank0]: engine = cls(
[rank0]: File "/root/vllm/vllm/engine/async_llm_engine.py", line 328, in __init__
[rank0]: self.engine = self._init_engine(*args, **kwargs)
[rank0]: File "/root/vllm/vllm/engine/async_llm_engine.py", line 450, in _init_engine
[rank0]: return engine_class(*args, **kwargs)
[rank0]: File "/root/vllm/vllm/engine/llm_engine.py", line 163, in __init__
[rank0]: self.model_executor = executor_class(
[rank0]: File "/root/vllm/vllm/executor/executor_base.py", line 41, in __init__
[rank0]: self._init_executor()
[rank0]: File "/root/vllm/vllm/executor/gpu_executor.py", line 24, in _init_executor
[rank0]: self.driver_worker.load_model()
[rank0]: File "/root/vllm/vllm/worker/worker.py", line 121, in load_model
[rank0]: self.model_runner.load_model()
[rank0]: File "/root/vllm/vllm/worker/model_runner.py", line 133, in load_model
[rank0]: self.model = get_model(
[rank0]: File "/root/vllm/vllm/model_executor/model_loader/__init__.py", line 21, in get_model
[rank0]: return loader.load_model(model_config=model_config,
[rank0]: File "/root/vllm/vllm/model_executor/model_loader/loader.py", line 227, in load_model
[rank0]: model = _initialize_model(model_config, self.load_config,
[rank0]: File "/root/vllm/vllm/model_executor/model_loader/loader.py", line 90, in _initialize_model
[rank0]: return model_class(config=model_config.hf_config,
[rank0]: TypeError: DeepseekV2ForCausalLM.__init__() got an unexpected keyword argument 'cache_config'
How to solve it?
(deepseek) ailearn@gpts:/data/sdd/models$ cd /data/sdd/models/ ; CUDA_VISIBLE_DEVICES=0,1,2,3 python -m vllm.entrypoints.openai.api_server --gpu-memory-utilization 0.99 --max-model-len 1024 --model DeepSeek-V2-Lite-Chat --enforce-eager --trust-remote-code --tensor-parallel-size 4 --host 0.0.0.0 --port 8008
2024-05-22 23:31:01,969 INFO worker.py:1749 -- Started a local Ray instance.
INFO 05-22 23:31:03 llm_engine.py:100] Initializing an LLM engine (v0.4.2) with config: model='DeepSeek-V2-Lite-Chat', speculative_config=None, tokenizer='DeepSeek-V2-Lite-Chat', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=1024, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=DeepSeek-V2-Lite-Chat)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
(RayWorkerWrapper pid=1195524) INFO 05-22 23:31:14 selector.py:81] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. pip install vllm-flash-attn for better performance.
(RayWorkerWrapper pid=1195524) INFO 05-22 23:31:14 selector.py:32] Using XFormers backend.
INFO 05-22 23:31:14 selector.py:81] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. pip install vllm-flash-attn for better performance.
INFO 05-22 23:31:14 selector.py:32] Using XFormers backend.
INFO 05-22 23:31:16 utils.py:638] Found nccl from library /home/ailearn/.config/vllm/nccl/cu11/libnccl.so.2.18.1
(RayWorkerWrapper pid=1195524) INFO 05-22 23:31:16 utils.py:638] Found nccl from library /home/ailearn/.config/vllm/nccl/cu11/libnccl.so.2.18.1
INFO 05-22 23:31:16 pynccl.py:65] vLLM is using nccl==2.18.1
(RayWorkerWrapper pid=1195524) INFO 05-22 23:31:16 pynccl.py:65] vLLM is using nccl==2.18.1
WARNING 05-22 23:31:16 custom_all_reduce.py:69] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(RayWorkerWrapper pid=1195524) WARNING 05-22 23:31:16 custom_all_reduce.py:69] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
Cache shape torch.Size([163840, 64])
(RayWorkerWrapper pid=1195524) Cache shape torch.Size([163840, 64])
INFO 05-22 23:31:21 model_runner.py:167] Loading model weights took 7.3840 GB
(RayWorkerWrapper pid=1195949) INFO 05-22 23:31:21 model_runner.py:167] Loading model weights took 7.3840 GB
(RayWorkerWrapper pid=1195949) INFO 05-22 23:31:14 selector.py:81] Cannot use FlashAttention-2 backend because the vllm_flash_attn package is not found. pip install vllm-flash-attn for better performance. [repeated 2x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(RayWorkerWrapper pid=1195949) INFO 05-22 23:31:14 selector.py:32] Using XFormers backend. [repeated 2x across cluster]
(RayWorkerWrapper pid=1195949) INFO 05-22 23:31:16 utils.py:638] Found nccl from library /home/ailearn/.config/vllm/nccl/cu11/libnccl.so.2.18.1 [repeated 2x across cluster]
(RayWorkerWrapper pid=1195949) INFO 05-22 23:31:16 pynccl.py:65] vLLM is using nccl==2.18.1 [repeated 2x across cluster]
(RayWorkerWrapper pid=1195949) WARNING 05-22 23:31:16 custom_all_reduce.py:69] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [repeated 2x across cluster]
(RayWorkerWrapper pid=1195949) Cache shape torch.Size([163840, 64]) [repeated 2x across cluster]
ERROR 05-22 23:31:23 worker_base.py:145] Error executing method determine_num_available_blocks. This might cause deadlock in distributed execution.
ERROR 05-22 23:31:23 worker_base.py:145] Traceback (most recent call last):
ERROR 05-22 23:31:23 worker_base.py:145] File "/data/sdd/deploy/deepseek/vllm/vllm/worker/worker_base.py", line 137, in execute_method
ERROR 05-22 23:31:23 worker_base.py:145] return executor(*args, **kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/home/ailearn/.conda/envs/deepseek/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR 05-22 23:31:23 worker_base.py:145] return func(*args, **kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/data/sdd/deploy/deepseek/vllm/vllm/worker/worker.py", line 138, in determine_num_available_blocks
ERROR 05-22 23:31:23 worker_base.py:145] self.model_runner.profile_run()
ERROR 05-22 23:31:23 worker_base.py:145] File "/home/ailearn/.conda/envs/deepseek/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR 05-22 23:31:23 worker_base.py:145] return func(*args, **kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/data/sdd/deploy/deepseek/vllm/vllm/worker/model_runner.py", line 875, in profile_run
ERROR 05-22 23:31:23 worker_base.py:145] self.execute_model(seqs, kv_caches)
ERROR 05-22 23:31:23 worker_base.py:145] File "/home/ailearn/.conda/envs/deepseek/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
ERROR 05-22 23:31:23 worker_base.py:145] return func(*args, **kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/data/sdd/deploy/deepseek/vllm/vllm/worker/model_runner.py", line 793, in execute_model
ERROR 05-22 23:31:23 worker_base.py:145] hidden_states = model_executable(**execute_model_kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/home/ailearn/.conda/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
ERROR 05-22 23:31:23 worker_base.py:145] return self._call_impl(*args, **kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/home/ailearn/.conda/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
ERROR 05-22 23:31:23 worker_base.py:145] return forward_call(*args, **kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/data/sdd/deploy/deepseek/vllm/vllm/model_executor/models/deepseek_v2.py", line 470, in forward
ERROR 05-22 23:31:23 worker_base.py:145] hidden_states = self.model(input_ids, positions, kv_caches,
ERROR 05-22 23:31:23 worker_base.py:145] File "/home/ailearn/.conda/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
ERROR 05-22 23:31:23 worker_base.py:145] return self._call_impl(*args, **kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/home/ailearn/.conda/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
ERROR 05-22 23:31:23 worker_base.py:145] return forward_call(*args, **kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/data/sdd/deploy/deepseek/vllm/vllm/model_executor/models/deepseek_v2.py", line 441, in forward
ERROR 05-22 23:31:23 worker_base.py:145] hidden_states, residual = layer(positions, hidden_states,
ERROR 05-22 23:31:23 worker_base.py:145] File "/home/ailearn/.conda/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
ERROR 05-22 23:31:23 worker_base.py:145] return self._call_impl(*args, **kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/home/ailearn/.conda/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
ERROR 05-22 23:31:23 worker_base.py:145] return forward_call(*args, **kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/data/sdd/deploy/deepseek/vllm/vllm/model_executor/models/deepseek_v2.py", line 401, in forward
ERROR 05-22 23:31:23 worker_base.py:145] hidden_states = self.mlp(hidden_states)
ERROR 05-22 23:31:23 worker_base.py:145] File "/home/ailearn/.conda/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
ERROR 05-22 23:31:23 worker_base.py:145] return self._call_impl(*args, **kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/home/ailearn/.conda/envs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
ERROR 05-22 23:31:23 worker_base.py:145] return forward_call(*args, **kwargs)
ERROR 05-22 23:31:23 worker_base.py:145] File "/data/sdd/deploy/deepseek/vllm/vllm/model_executor/models/deepseek_v2.py", line 163, in forward
ERROR 05-22 23:31:23 worker_base.py:145] final_hidden_states = fused_experts(
ERROR 05-22 23:31:23 worker_base.py:145] File "/data/sdd/deploy/deepseek/vllm/vllm/model_executor/layers/fused_moe/fused_moe.py", line 455, in fused_experts
ERROR 05-22 23:31:23 worker_base.py:145] invoke_fused_moe_kernel(hidden_states,
ERROR 05-22 23:31:23 worker_base.py:145] File "/data/sdd/deploy/deepseek/vllm/vllm/model_executor/layers/fused_moe/fused_moe.py", line 246, in invoke_fused_moe_kernel
ERROR 05-22 23:31:23 worker_base.py:145] fused_moe_kernel[grid](
ERROR 05-22 23:31:23 worker_base.py:145] File "/home/ailearn/.conda/envs/deepseek/lib/python3.10/site-packages/triton/runtime/jit.py", line 167, in
I follow the file change to change the file ,but when I use the below code :
python -m vllm.entrypoints.openai.api_server --model /root/DeepSeek-V2-Chat --trust-remote-codethere is a error happen:INFO 05-22 06:30:51 llm_engine.py:103] Initializing an LLM engine (v0.4.2) with config: model='/root/DeepSeek-V2-Chat', speculative_config=None, tokenizer='/root/DeepSeek-V2-Chat', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=163840, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/root/DeepSeek-V2-Chat) Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. INFO 05-22 06:30:52 selector.py:44] Using FlashAttention-2 backend. [rank0]: Traceback (most recent call last): [rank0]: File "/opt/conda/envs/deepseek/lib/python3.10/runpy.py", line 196, in _run_module_as_main [rank0]: return _run_code(code, main_globals, None, [rank0]: File "/opt/conda/envs/deepseek/lib/python3.10/runpy.py", line 86, in _run_code [rank0]: exec(code, run_globals) [rank0]: File "/root/vllm/vllm/entrypoints/openai/api_server.py", line 186, in <module> [rank0]: engine = AsyncLLMEngine.from_engine_args( [rank0]: File "/root/vllm/vllm/engine/async_llm_engine.py", line 374, in from_engine_args [rank0]: engine = cls( [rank0]: File "/root/vllm/vllm/engine/async_llm_engine.py", line 328, in __init__ [rank0]: self.engine = self._init_engine(*args, **kwargs) [rank0]: File "/root/vllm/vllm/engine/async_llm_engine.py", line 450, in _init_engine [rank0]: return engine_class(*args, **kwargs) [rank0]: File "/root/vllm/vllm/engine/llm_engine.py", line 163, in __init__ [rank0]: self.model_executor = executor_class( [rank0]: File "/root/vllm/vllm/executor/executor_base.py", line 41, in __init__ [rank0]: self._init_executor() [rank0]: File "/root/vllm/vllm/executor/gpu_executor.py", line 24, in _init_executor [rank0]: self.driver_worker.load_model() [rank0]: File "/root/vllm/vllm/worker/worker.py", line 121, in load_model [rank0]: self.model_runner.load_model() [rank0]: File "/root/vllm/vllm/worker/model_runner.py", line 133, in load_model [rank0]: self.model = get_model( [rank0]: File "/root/vllm/vllm/model_executor/model_loader/__init__.py", line 21, in get_model [rank0]: return loader.load_model(model_config=model_config, [rank0]: File "/root/vllm/vllm/model_executor/model_loader/loader.py", line 227, in load_model [rank0]: model = _initialize_model(model_config, self.load_config, [rank0]: File "/root/vllm/vllm/model_executor/model_loader/loader.py", line 90, in _initialize_model [rank0]: return model_class(config=model_config.hf_config, [rank0]: TypeError: DeepseekV2ForCausalLM.__init__() got an unexpected keyword argument 'cache_config'How to solve it?
you can see that cache_config was recently added in this commit:
https://github.com/seungduk-yanolja/vllm-deepseek/commit/0fca3cdcf265cd375bca684d951702b6b7adf65a
I made a fix with recent changes on vLLM. https://github.com/seungduk-yanolja/vllm-deepseek
Assuming you have an 8xH100 machine, to run,
python -m vllm.entrypoints.openai.api_server --model deepseek-ai/DeepSeek-V2-Chat -tp 8 --served-model-name deeps
eek --trust-remote-code --max-model-len=2800 --enforce-eager
I made a fix with recent changes on vLLM. https://github.com/seungduk-yanolja/vllm-deepseek
Assuming you have an 8xH100 machine, to run,
python -m vllm.entrypoints.openai.api_server --model deepseek-ai/DeepSeek-V2-Chat -tp 8 --served-model-name deeps eek --trust-remote-code --max-model-len=2800 --enforce-eager
Thank you for your help. Can I use this option with A100*8 GPU?
I made a fix with recent changes on vLLM. https://github.com/seungduk-yanolja/vllm-deepseek Assuming you have an 8xH100 machine, to run,
python -m vllm.entrypoints.openai.api_server --model deepseek-ai/DeepSeek-V2-Chat -tp 8 --served-model-name deeps eek --trust-remote-code --max-model-len=2800 --enforce-eagerThank you for your help. Can I use this option with A100*8 GPU?
I think so. If you need more memory, vLLM will complain about max_model_len so you can decrease it to make it run.
I made a fix with recent changes on vLLM. https://github.com/seungduk-yanolja/vllm-deepseek Assuming you have an 8xH100 machine, to run,
python -m vllm.entrypoints.openai.api_server --model deepseek-ai/DeepSeek-V2-Chat -tp 8 --served-model-name deeps eek --trust-remote-code --max-model-len=2800 --enforce-eagerThank you for your help. Can I use this option with A100*8 GPU?
I think so. If you need more memory, vLLM will complain about
max_model_lenso you can decrease it to make it run.
Thanks a lot! I am going to test it and share results.
I made a fix with recent changes on vLLM. https://github.com/seungduk-yanolja/vllm-deepseek Assuming you have an 8xH100 machine, to run,
python -m vllm.entrypoints.openai.api_server --model deepseek-ai/DeepSeek-V2-Chat -tp 8 --served-model-name deeps eek --trust-remote-code --max-model-len=2800 --enforce-eagerThank you for your help. Can I use this option with A100*8 GPU?
I think so. If you need more memory, vLLM will complain about
max_model_lenso you can decrease it to make it run.Thanks a lot! I am going to test it and share results.
Thank u for ur attempt, so what's the result?
@zwd003 I did the refactoring of the MoE code for you, can you look into the other comments I just added?
OK
I follow the file change to change the file ,but when I use the below code :
python -m vllm.entrypoints.openai.api_server --model /root/DeepSeek-V2-Chat --trust-remote-codethere is a error happen:INFO 05-22 06:30:51 llm_engine.py:103] Initializing an LLM engine (v0.4.2) with config: model='/root/DeepSeek-V2-Chat', speculative_config=None, tokenizer='/root/DeepSeek-V2-Chat', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=163840, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/root/DeepSeek-V2-Chat) Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. INFO 05-22 06:30:52 selector.py:44] Using FlashAttention-2 backend. [rank0]: Traceback (most recent call last): [rank0]: File "/opt/conda/envs/deepseek/lib/python3.10/runpy.py", line 196, in _run_module_as_main [rank0]: return _run_code(code, main_globals, None, [rank0]: File "/opt/conda/envs/deepseek/lib/python3.10/runpy.py", line 86, in _run_code [rank0]: exec(code, run_globals) [rank0]: File "/root/vllm/vllm/entrypoints/openai/api_server.py", line 186, in <module> [rank0]: engine = AsyncLLMEngine.from_engine_args( [rank0]: File "/root/vllm/vllm/engine/async_llm_engine.py", line 374, in from_engine_args [rank0]: engine = cls( [rank0]: File "/root/vllm/vllm/engine/async_llm_engine.py", line 328, in __init__ [rank0]: self.engine = self._init_engine(*args, **kwargs) [rank0]: File "/root/vllm/vllm/engine/async_llm_engine.py", line 450, in _init_engine [rank0]: return engine_class(*args, **kwargs) [rank0]: File "/root/vllm/vllm/engine/llm_engine.py", line 163, in __init__ [rank0]: self.model_executor = executor_class( [rank0]: File "/root/vllm/vllm/executor/executor_base.py", line 41, in __init__ [rank0]: self._init_executor() [rank0]: File "/root/vllm/vllm/executor/gpu_executor.py", line 24, in _init_executor [rank0]: self.driver_worker.load_model() [rank0]: File "/root/vllm/vllm/worker/worker.py", line 121, in load_model [rank0]: self.model_runner.load_model() [rank0]: File "/root/vllm/vllm/worker/model_runner.py", line 133, in load_model [rank0]: self.model = get_model( [rank0]: File "/root/vllm/vllm/model_executor/model_loader/__init__.py", line 21, in get_model [rank0]: return loader.load_model(model_config=model_config, [rank0]: File "/root/vllm/vllm/model_executor/model_loader/loader.py", line 227, in load_model [rank0]: model = _initialize_model(model_config, self.load_config, [rank0]: File "/root/vllm/vllm/model_executor/model_loader/loader.py", line 90, in _initialize_model [rank0]: return model_class(config=model_config.hf_config, [rank0]: TypeError: DeepseekV2ForCausalLM.__init__() got an unexpected keyword argument 'cache_config'How to solve it?
i also met this error . And i find that DeepseekV2ForCausalLM do not have the cache_config args . How can i fix this. Thanks!
same problem [rank0]: TypeError: DeepseekV2ForCausalLM.init() got an unexpected keyword argument 'cache_config'
