vllm-project
[Bug]: V1 engine fails with offline batched inference code in V0 engine
Your current environment
The output of `python collect_env.py`
INFO 02-08 09:51:29 __init__.py:190] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A
OS: Alibaba Group Enterprise Linux Server 7.2 (Paladin) (x86_64)
GCC version: (GCC) 10.5.0
Clang version: Could not collect
CMake version: version 3.26.4
Libc version: glibc-2.32
Python version: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.9.151-015.ali3000.alios7.x86_64-x86_64-with-glibc2.32
Is CUDA available: True
CUDA runtime version: 12.6.85
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
Nvidia driver version: 470.82.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 106
Model name: Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz
Stepping: 6
CPU MHz: 3499.859
CPU max MHz: 3500.0000
CPU min MHz: 800.0000
BogoMIPS: 5800.00
Virtualization: VT-x
L1d cache: 48K
L1i cache: 32K
L2 cache: 1280K
L3 cache: 49152K
NUMA node0 CPU(s): 0-31,64-95
NUMA node1 CPU(s): 32-63,96-127
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq pconfig flush_l1d arch_capabilities
Versions of relevant libraries:
[pip3] flake8==7.1.1
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] sentence-transformers==3.4.1
[pip3] torch==2.5.1
[pip3] torchaudio==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.48.2
[pip3] triton==3.1.0
[conda] numpy 1.26.4 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.5.147 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.6.3 pypi_0 pypi
[conda] nvidia-ml-py 12.560.30 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.21.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.4.127 pypi_0 pypi
[conda] pyzmq 26.2.0 pypi_0 pypi
[conda] sentence-transformers 3.4.1 pypi_0 pypi
[conda] torch 2.5.1 pypi_0 pypi
[conda] torchaudio 2.5.1 pypi_0 pypi
[conda] torchvision 0.20.1 pypi_0 pypi
[conda] transformers 4.48.2 pypi_0 pypi
[conda] triton 3.1.0 pypi_0 pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.7.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 CPU Affinity NUMA Affinity
GPU0 X NV12 0-31,64-95 0
GPU1 NV12 X 0-31,64-95 0
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NVIDIA_VISIBLE_DEVICES=GPU-5294a6a7-80ee-6c4f-6dee-f115cffa4c48,GPU-a22ea803-549c-7751-b818-7786ff518963
LD_LIBRARY_PATH=:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64:/opt/conda/lib/python3.10/site-packages/aistudio_common/reader/libs/:/opt/taobao/java/jre/lib/amd64/server/:/usr/local/cuda/lib64
NVIDIA_DRIVER_CAPABILITIES=all
NCCL_NVLS_ENABLE=0
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY
🐛 Describe the bug
I'm inferencing multiple prompts (inconsistent length) with local vllm in python code. If changed to VLLM V1 engine, it fails. If reversed to V0 engine, it can finish without any error. The startup code is simplified like this:
model_path = "/nfs/llm_models/DeepSeek-R1-Distill-Qwen-32B"
stop_token_ids = [151645, 151643]
llm = LLM(model=model_path, tokenizer=None, tensor_parallel_size=torch.cuda.device_count(), trust_remote_code=True, gpu_memory_utilization=0.8)
tokenizer = AutoTokenizer.from_pretrained(model_path)
input_texts = []
for one_prompt in prompts:
messages = [
{"role": "system", "content": ""},
{"role": "user", "content": one_prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
input_texts.append(text)
output_strs = llm.generate(input_texts, sampling_params, use_tqdm=True)
Here's the log and traceback:
Processed prompts: 0%| | 4/3549 [00:50<9:04:40, 9.22s/it, est. speed input: 48.56 toks/s, output: 12.39 toks/s](VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] WorkerProc hit an exception: %s
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] Traceback (most recent call last):
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 370, in worker_busy_loop
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] output = func(*args, **kwargs)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] return func(*args, **kwargs)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 236, in execute_model
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] output = self.model_runner.execute_model(scheduler_output)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] return func(*args, **kwargs)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 804, in execute_model
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] sampler_output = self.model.sample(
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 505, in sample
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] next_tokens = self.sampler(logits, sampling_metadata)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/sample/sampler.py", line 46, in forward
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] logits = self.apply_penalties(logits, sampling_metadata)
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/sample/sampler.py", line 130, in apply_penalties
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] logits = apply_all_penalties(
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/sample/ops/penalties.py", line 42, in apply_all_penalties
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] return apply_penalties(logits, prompt_token_ids, output_tokens_t,
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/utils.py", line 44, in apply_penalties
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] _, prompt_mask = get_token_bin_counts_and_mask(prompt_tokens_tensor,
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/utils.py", line 18, in get_token_bin_counts_and_mask
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] bin_counts.scatter_add_(1, tokens, torch.ones_like(tokens))
(VllmWorker rank=0 pid=9923) ERROR 02-08 09:31:59 multiproc_executor.py:374] RuntimeError: Expected index [668, 651] to be smaller than self [667, 152065] apart from dimension 1 and to be smaller size than src [668, 651]
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] WorkerProc hit an exception: %s
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] Traceback (most recent call last):
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 370, in worker_busy_loop
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] output = func(*args, **kwargs)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] return func(*args, **kwargs)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 236, in execute_model
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] output = self.model_runner.execute_model(scheduler_output)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] return func(*args, **kwargs)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 804, in execute_model
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] sampler_output = self.model.sample(
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 505, in sample
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] next_tokens = self.sampler(logits, sampling_metadata)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/sample/sampler.py", line 46, in forward
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] logits = self.apply_penalties(logits, sampling_metadata)
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/sample/sampler.py", line 130, in apply_penalties
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] logits = apply_all_penalties(
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/sample/ops/penalties.py", line 42, in apply_all_penalties
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] return apply_penalties(logits, prompt_token_ids, output_tokens_t,
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/utils.py", line 44, in apply_penalties
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] _, prompt_mask = get_token_bin_counts_and_mask(prompt_tokens_tensor,
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/utils.py", line 18, in get_token_bin_counts_and_mask
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] bin_counts.scatter_add_(1, tokens, torch.ones_like(tokens))
(VllmWorker rank=1 pid=9982) ERROR 02-08 09:31:59 multiproc_executor.py:374] RuntimeError: Expected index [668, 651] to be smaller than self [667, 152065] apart from dimension 1 and to be smaller size than src [668, 651]
ERROR 02-08 09:31:59 core.py:210] EngineCore hit an exception: Traceback (most recent call last):
ERROR 02-08 09:31:59 core.py:210] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 203, in run_engine_core
ERROR 02-08 09:31:59 core.py:210] engine_core.run_busy_loop()
ERROR 02-08 09:31:59 core.py:210] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 243, in run_busy_loop
ERROR 02-08 09:31:59 core.py:210] outputs = self.step()
ERROR 02-08 09:31:59 core.py:210] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 129, in step
ERROR 02-08 09:31:59 core.py:210] output = self.model_executor.execute_model(scheduler_output)
ERROR 02-08 09:31:59 core.py:210] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 77, in execute_model
ERROR 02-08 09:31:59 core.py:210] output = self.collective_rpc("execute_model",
ERROR 02-08 09:31:59 core.py:210] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 133, in collective_rpc
ERROR 02-08 09:31:59 core.py:210] raise e
ERROR 02-08 09:31:59 core.py:210] File "/opt/conda/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 122, in collective_rpc
ERROR 02-08 09:31:59 core.py:210] raise result
Before submitting a new issue...
- [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
Encountered the same problem. But I'm using vllm serve to deploy DeepSeek-R1-AWQ.
Environment
- Image: cuda:12.6.0-cudnn-devel-ubuntu22.04
- GPUs: A800 x 8
- Python 3.10
- vLLM 0.7.2
- torch 2.5.1
Deploy scripts
#!/bin/bash
export VLLM_USE_V1=1
export VLLM_CUDA_MEM_ALIGN_KV_CACHE=1
export VLLM_USE_TRITON_AWQ=1
vllm serve \
/path/to/DeepSeek-R1-AWQ \
--served-model-name "DeepSeek-R1" \
--trust-remote-code \
--host "0.0.0.0" \
--port 20000 \
--max-model-len 16384 \
--gpu-memory-utilization 0.95 \
--tensor-parallel-size 8 \
--swap-space 32 \
--quantization moe_wna16 \
--enforce-eager \
--enable-reasoning \
--reasoning-parser deepseek_r1
After reading the source code, I have the following findings.
V0 Implement
Function get_token_bin_counts_and_mask(vllm.model_executor.layers.utils.py, line8), the shape of bin_counts is [num_seq, vocab_size + 1]
The num_seqs is logits.shape[0]
The tokens is sampling_tensors.prompt_tokens, created from SamplingTensors.from_sampling_metadata. The prompt_tokens is not empty only when do_penalties is True, and the shape of prompt_tokens is [len(sampling_metadata.seq_groups), ...]
The sampling_metadata is vllm.model_executor.sampling_metadata.SamplingMetadata
V1 Implement
The sampling_metadata is vllm.v1.sample.metadata.SamplingMetadata
The tokens is sampling_metadata.prompt_token_ids, which created from vllm.v1.worker.gpu_input_batch.InputBatch._make_prompt_token_ids_tensor. The shape of it is [InputBatch.num_reqs, max_prompt_len]
The InputBatch.num_reqs is property, equals len(InputBatch.req_id_to_index)
Could it be that the inconsistency between logits.shape[0] and len(InputBatch.req_id_to_index) is causing this issue?
More precisely, could it be that during the sampling phase, new requests are entering, leading to inconsistency in seqs?
I can apply a temporary fix by directly modifying the code to address this error. However, I am uncertain about the correctness of this approach, so this suggestion is provided for reference only.
https://github.com/vllm-project/vllm/blob/3243158336d377c8aced151722b5f8bbff2f905d/vllm/v1/sample/ops/penalties.py#L28
def apply_all_penalties(
logits: torch.Tensor,
prompt_token_ids: torch.Tensor,
presence_penalties: torch.Tensor,
frequency_penalties: torch.Tensor,
repetition_penalties: torch.Tensor,
output_token_ids: List[List[int]],
) -> torch.Tensor:
"""
Applies presence, frequency and repetition penalties to the logits.
"""
num_seqs, vocab_size = logits.shape
output_tokens_t = _convert_to_tensors(output_token_ids, vocab_size,
logits.device)
return apply_penalties(logits, prompt_token_ids[:num_seqs], output_tokens_t,
presence_penalties, frequency_penalties,
repetition_penalties)
Can you share a working repro script:
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
model_path = "meta-llama/Meta-Llama-3.1-8B-Instruct"
stop_token_ids = [11469, 11]
llm = LLM(model=model_path,
tensor_parallel_size=1,
trust_remote_code=True,
gpu_memory_utilization=0.8)
tokenizer = AutoTokenizer.from_pretrained(model_path)
prompts = [
"Hello. Who is the best company supporting vLLM?",
"Who is the market leader in enterprise Linux and Kubernetes?"
]
input_texts = []
for one_prompt in prompts:
messages = [
{"role": "system", "content": ""},
{"role": "user", "content": one_prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
input_texts.append(text)
sampling_params = SamplingParams(stop_token_ids=stop_token_ids,
max_tokens=100,
temperature=0)
outputs = llm.generate(input_texts, sampling_params, use_tqdm=True)
print(outputs)
we observed a same issue when deploy llama 3.3 ERROR 02-17 01:53:55 core.py:210] EngineCore hit an exception: Traceback (most recent call last): ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 203, in run_engine_core ERROR 02-17 01:53:55 core.py:210] engine_core.run_busy_loop() ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 243, in run_busy_loop ERROR 02-17 01:53:55 core.py:210] outputs = self.step() ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 129, in step ERROR 02-17 01:53:55 core.py:210] output = self.model_executor.execute_model(scheduler_output) ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 77, in execute_model ERROR 02-17 01:53:55 core.py:210] output = self.collective_rpc("execute_model", ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 51, in collective_rpc ERROR 02-17 01:53:55 core.py:210] answer = run_method(self.driver_worker, method, args, kwargs) ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2220, in run_method ERROR 02-17 01:53:55 core.py:210] return func(*args, **kwargs) ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 02-17 01:53:55 core.py:210] return func(*args, **kwargs) ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 236, in execute_model ERROR 02-17 01:53:55 core.py:210] output = self.model_runner.execute_model(scheduler_output) ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 02-17 01:53:55 core.py:210] return func(*args, **kwargs) ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 804, in execute_model ERROR 02-17 01:53:55 core.py:210] sampler_output = self.model.sample( ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 557, in sample ERROR 02-17 01:53:55 core.py:210] next_tokens = self.sampler(logits, sampling_metadata) ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl ERROR 02-17 01:53:55 core.py:210] return self._call_impl(*args, **kwargs) ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl ERROR 02-17 01:53:55 core.py:210] return forward_call(*args, **kwargs) ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/sampler.py", line 46, in forward ERROR 02-17 01:53:55 core.py:210] logits = self.apply_penalties(logits, sampling_metadata) ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/sampler.py", line 130, in apply_penalties ERROR 02-17 01:53:55 core.py:210] logits = apply_all_penalties( ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/sample/ops/penalties.py", line 42, in apply_all_penalties ERROR 02-17 01:53:55 core.py:210] return apply_penalties(logits, prompt_token_ids, output_tokens_t, ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/utils.py", line 44, in apply_penalties ERROR 02-17 01:53:55 core.py:210] , prompt_mask = get_token_bin_counts_and_mask(prompt_tokens_tensor, ERROR 02-17 01:53:55 core.py:210] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 02-17 01:53:55 core.py:210] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/utils.py", line 18, in get_token_bin_counts_and_mask ERROR 02-17 01:53:55 core.py:210] bin_counts.scatter_add(1, tokens, torch.ones_like(tokens)) ERROR 02-17 01:53:55 core.py:210] RuntimeError: Expected index [4, 1193] to be smaller than self [3, 128257] apart from dimension 1 and to be smaller size than src [4, 1193]
@Fy- I think the issue is fixed with latest code. https://github.com/vllm-project/vllm/issues/13076 I bake the main branch and no issues
This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!