vllm
vllm copied to clipboard
Published
20 hours ago •
vllm-project
vllm-project
[Bug]: assert len(self._async_stopped) == 0
Your current environment
The output of `python collect_env.py`
# For security purposes, please feel free to check the contents of collect_env.py before running it.
python collect_env.py
--2024-09-27 03:02:25-- https://raw.githubusercontent.com/vllm-project/vllm/main/collect_env.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.108.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.
Unable to establish SSL connection.
python: can't open file '/home/corvo/collect_env.py': [Errno 2] No such file or directory
corvo@llmpfs-mistral-large-vllmd-0-0:~$ cd /models/
corvo@llmpfs-mistral-large-vllmd-0-0:/models$ python collect_env.py
Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.2
Libc version: glibc-2.35
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.10.223-212.873.amzn2.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
Nvidia driver version: 550.90.07
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 192
On-line CPU(s) list: 0-191
Vendor ID: AuthenticAMD
Model name: AMD EPYC 7R13 Processor
CPU family: 25
Model: 1
Thread(s) per core: 2
Core(s) per socket: 48
Socket(s): 2
Stepping: 1
BogoMIPS: 5299.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 3 MiB (96 instances)
L1i cache: 3 MiB (96 instances)
L2 cache: 48 MiB (96 instances)
L3 cache: 384 MiB (12 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-47,96-143
NUMA node1 CPU(s): 48-95,144-191
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET, no microcode
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] onnx==1.16.0
[pip3] optree==0.11.0
[pip3] pytorch-quantization==2.1.2
[pip3] pytorch-triton==3.0.0+989adb9a2
[pip3] pyzmq==26.0.3
[pip3] torch==2.4.0
[pip3] torch-tensorrt==2.4.0a0
[pip3] torchvision==0.19.0
[pip3] transformers==4.44.2
[pip3] triton==3.0.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.1.post2@156403f983a2922fc2d5dc9da54be2cd474211e0
vLLM Build Flags:
CUDA Archs: 5.2 6.0 6.1 7.0 7.2 7.5 8.0 8.6 8.7 9.0+PTX; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 48-95,144-191 1 N/A
GPU1 NV18 X NV18 NV18 48-95,144-191 1 N/A
GPU2 NV18 NV18 X NV18 48-95,144-191 1 N/A
GPU3 NV18 NV18 NV18 X 48-95,144-191 1 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
Model Input Dumps
No response
🐛 Describe the bug
We are wrapping around vllm's AsyncLLMEngine, code can be simplified as below.
engine = AsyncLLMEngine.from_engine_args(self.engine_args)
and then use engine to handle request
def request_handler():
engine.generate(
inputs=prompt,
sampling_params=sampling_params,
request_id=request_id,
lora_request=lora_request,
priority=priority,
)
The error is
INFO 09-27 03:03:51 llm_engine.py:223] Initializing an LLM engine (v0.6.1.post2) with config: model='/models/mistral-large2', speculative_config=None, tokenizer='/models/mistral-large2', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/models/mistral-large2, use_v2_block_manager=False, num_scheduler_steps=1, enable_prefix_caching=False, use_async_output_proc=True)
WARNING 09-27 03:03:51 multiproc_gpu_executor.py:56] Reducing Torch parallelism from 96 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 09-27 03:03:51 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
(VllmWorkerProcess pid=180) INFO 09-27 03:03:51 selector.py:116] Using XFormers backend.
(VllmWorkerProcess pid=215) INFO 09-27 03:03:51 selector.py:116] Using XFormers backend.
INFO 09-27 03:03:51 selector.py:116] Using XFormers backend.
(VllmWorkerProcess pid=250) INFO 09-27 03:03:51 selector.py:116] Using XFormers backend.
(VllmWorkerProcess pid=180) /home/corvo/.local/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=180) @torch.library.impl_abstract("xformers_flash::flash_fwd")
(VllmWorkerProcess pid=215) /home/corvo/.local/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=215) @torch.library.impl_abstract("xformers_flash::flash_fwd")
/home/corvo/.local/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_fwd")
(VllmWorkerProcess pid=250) /home/corvo/.local/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=250) @torch.library.impl_abstract("xformers_flash::flash_fwd")
(VllmWorkerProcess pid=180) /home/corvo/.local/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=180) @torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=215) /home/corvo/.local/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=215) @torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=250) /home/corvo/.local/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=250) @torch.library.impl_abstract("xformers_flash::flash_bwd")
/home/corvo/.local/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=250) INFO 09-27 03:03:54 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=180) INFO 09-27 03:03:54 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=215) INFO 09-27 03:03:54 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=215) INFO 09-27 03:03:54 utils.py:981] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=180) INFO 09-27 03:03:54 utils.py:981] Found nccl from library libnccl.so.2
INFO 09-27 03:03:54 utils.py:981] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=215) (VllmWorkerProcess pid=180) INFO 09-27 03:03:54 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 09-27 03:03:54 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 09-27 03:03:54 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=250) INFO 09-27 03:03:54 utils.py:981] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=250) INFO 09-27 03:03:54 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=215) INFO 09-27 03:03:56 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/corvo/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3.json
(VllmWorkerProcess pid=250) INFO 09-27 03:03:56 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/corvo/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3.json
INFO 09-27 03:03:56 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/corvo/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3.json
(VllmWorkerProcess pid=180) INFO 09-27 03:03:56 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/corvo/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3.json
INFO 09-27 03:03:56 shm_broadcast.py:235] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7fc7022a4760>, local_subscribe_port=49977, remote_subscribe_port=None)
INFO 09-27 03:03:56 model_runner.py:999] Starting to load model /models/mistral-large2...
(VllmWorkerProcess pid=180) INFO 09-27 03:03:56 model_runner.py:999] Starting to load model /models/mistral-large2...
(VllmWorkerProcess pid=215) INFO 09-27 03:03:56 model_runner.py:999] Starting to load model /models/mistral-large2...
(VllmWorkerProcess pid=250) INFO 09-27 03:03:56 model_runner.py:999] Starting to load model /models/mistral-large2...
(VllmWorkerProcess pid=215) WARNING 09-27 03:03:56 fp8.py:47] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
(VllmWorkerProcess pid=180) WARNING 09-27 03:03:56 fp8.py:47] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
(VllmWorkerProcess pid=250) WARNING 09-27 03:03:56 fp8.py:47] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
WARNING 09-27 03:03:56 fp8.py:47] Detected fp8 checkpoint. Please note that the format is experimental and subject to change.
(VllmWorkerProcess pid=215) INFO 09-27 03:03:56 selector.py:116] Using XFormers backend.
(VllmWorkerProcess pid=180) INFO 09-27 03:03:56 selector.py:116] Using XFormers backend.
INFO 09-27 03:03:56 selector.py:116] Using XFormers backend.
(VllmWorkerProcess pid=250) INFO 09-27 03:03:56 selector.py:116] Using XFormers backend.
Loading safetensors checkpoint shards: 0% Completed | 0/26 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 4% Completed | 1/26 [00:00<00:03, 8.01it/s]
Loading safetensors checkpoint shards: 8% Completed | 2/26 [00:00<00:06, 3.56it/s]
Loading safetensors checkpoint shards: 12% Completed | 3/26 [00:00<00:08, 2.86it/s]
Loading safetensors checkpoint shards: 15% Completed | 4/26 [00:01<00:07, 2.77it/s]
Loading safetensors checkpoint shards: 19% Completed | 5/26 [00:01<00:07, 2.71it/s]
Loading safetensors checkpoint shards: 23% Completed | 6/26 [00:02<00:07, 2.60it/s]
Loading safetensors checkpoint shards: 27% Completed | 7/26 [00:02<00:07, 2.53it/s]
Loading safetensors checkpoint shards: 31% Completed | 8/26 [00:02<00:07, 2.44it/s]
Loading safetensors checkpoint shards: 35% Completed | 9/26 [00:03<00:07, 2.33it/s]
Loading safetensors checkpoint shards: 38% Completed | 10/26 [00:03<00:07, 2.26it/s]
Loading safetensors checkpoint shards: 42% Completed | 11/26 [00:04<00:06, 2.24it/s]
Loading safetensors checkpoint shards: 46% Completed | 12/26 [00:04<00:06, 2.12it/s]
Loading safetensors checkpoint shards: 50% Completed | 13/26 [00:05<00:06, 2.02it/s]
Loading safetensors checkpoint shards: 54% Completed | 14/26 [00:06<00:06, 1.91it/s]
Loading safetensors checkpoint shards: 58% Completed | 15/26 [00:06<00:06, 1.82it/s]
Loading safetensors checkpoint shards: 62% Completed | 16/26 [00:07<00:05, 1.77it/s]
Loading safetensors checkpoint shards: 65% Completed | 17/26 [00:07<00:05, 1.79it/s]
Loading safetensors checkpoint shards: 69% Completed | 18/26 [00:08<00:04, 1.79it/s]
Loading safetensors checkpoint shards: 73% Completed | 19/26 [00:08<00:03, 1.79it/s]
Loading safetensors checkpoint shards: 77% Completed | 20/26 [00:09<00:03, 1.77it/s]
Loading safetensors checkpoint shards: 81% Completed | 21/26 [00:10<00:02, 1.77it/s]
Loading safetensors checkpoint shards: 85% Completed | 22/26 [00:10<00:02, 1.78it/s]
Loading safetensors checkpoint shards: 88% Completed | 23/26 [00:11<00:01, 1.83it/s]
Loading safetensors checkpoint shards: 92% Completed | 24/26 [00:11<00:01, 1.85it/s]
Loading safetensors checkpoint shards: 96% Completed | 25/26 [00:12<00:00, 1.83it/s]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:12<00:00, 1.84it/s]
Loading safetensors checkpoint shards: 100% Completed | 26/26 [00:12<00:00, 2.04it/s]
(VllmWorkerProcess pid=215) INFO 09-27 03:04:09 model_runner.py:1010] Loading model weights took 28.7698 GB
(VllmWorkerProcess pid=180) INFO 09-27 03:04:10 model_runner.py:1010] Loading model weights took 28.7698 GB
(VllmWorkerProcess pid=250) INFO 09-27 03:04:10 model_runner.py:1010] Loading model weights took 28.7698 GB
INFO 09-27 03:04:10 model_runner.py:1010] Loading model weights took 28.7698 GB
INFO 09-27 03:04:12 distributed_gpu_executor.py:57] # GPU blocks: 32028, # CPU blocks: 2978
INFO 09-27 03:04:14 model_runner.py:1444] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 09-27 03:04:14 model_runner.py:1448] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=250) INFO 09-27 03:04:14 model_runner.py:1444] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=250) INFO 09-27 03:04:14 model_runner.py:1448] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=180) INFO 09-27 03:04:14 model_runner.py:1444] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=180) INFO 09-27 03:04:14 model_runner.py:1448] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=215) INFO 09-27 03:04:14 model_runner.py:1444] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=215) INFO 09-27 03:04:14 model_runner.py:1448] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=180) INFO 09-27 03:04:35 custom_all_reduce.py:240] Registering 6195 cuda graph addresses
(VllmWorkerProcess pid=250) INFO 09-27 03:04:35 custom_all_reduce.py:240] Registering 6195 cuda graph addresses
(VllmWorkerProcess pid=215) INFO 09-27 03:04:35 custom_all_reduce.py:240] Registering 6195 cuda graph addresses
INFO 09-27 03:04:35 custom_all_reduce.py:240] Registering 6195 cuda graph addresses
(VllmWorkerProcess pid=180) INFO 09-27 03:04:35 model_runner.py:1564] Graph capturing finished in 21 secs.
INFO 09-27 03:04:35 model_runner.py:1564] Graph capturing finished in 21 secs.
(VllmWorkerProcess pid=250) INFO 09-27 03:04:35 model_runner.py:1564] Graph capturing finished in 21 secs.
(VllmWorkerProcess pid=215) INFO 09-27 03:04:35 model_runner.py:1564] Graph capturing finished in 21 secs.
INFO 09-27 03:04:35 engine.py:125] Took 44.80 seconds to start vllm engine for model mistral-large2
INFO 09-27 03:04:57 metrics.py:352] Avg prompt throughput: 0.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 09-27 03:05:27 metrics.py:352] Avg prompt throughput: 0.1 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 09-27 03:05:43 metrics.py:352] Avg prompt throughput: 3.2 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 2 reqs, GPU KV cache usage: 46.5%, CPU KV cache usage: 0.0%.
ERROR 09-27 03:05:43 async_llm_engine.py:58] Engine background task failed
ERROR 09-27 03:05:43 async_llm_engine.py:58] Traceback (most recent call last):
ERROR 09-27 03:05:43 async_llm_engine.py:58] File "/home/corvo/vllm-project/vllm/engine/async_llm_engine.py", line 48, in _log_task_completion
ERROR 09-27 03:05:43 async_llm_engine.py:58] return_value = task.result()
ERROR 09-27 03:05:43 async_llm_engine.py:58] File "/home/corvo/vllm-project/vllm/engine/async_llm_engine.py", line 772, in run_engine_loop
ERROR 09-27 03:05:43 async_llm_engine.py:58] result = task.result()
ERROR 09-27 03:05:43 async_llm_engine.py:58] File "/home/corvo/vllm-project/vllm/engine/async_llm_engine.py", line 712, in engine_step
ERROR 09-27 03:05:43 async_llm_engine.py:58] request_outputs = await self.engine.step_async(virtual_engine)
ERROR 09-27 03:05:43 async_llm_engine.py:58] File "/home/corvo/vllm-project/vllm/engine/async_llm_engine.py", line 296, in step_async
ERROR 09-27 03:05:43 async_llm_engine.py:58] ) = self.scheduler[virtual_engine].schedule()
ERROR 09-27 03:05:43 async_llm_engine.py:58] File "/home/corvo/vllm-project/vllm/core/scheduler.py", line 1115, in schedule
ERROR 09-27 03:05:43 async_llm_engine.py:58] scheduler_outputs = self._schedule()
ERROR 09-27 03:05:43 async_llm_engine.py:58] File "/home/corvo/vllm-project/vllm/core/scheduler.py", line 1078, in _schedule
ERROR 09-27 03:05:43 async_llm_engine.py:58] return self._schedule_chunked_prefill()
ERROR 09-27 03:05:43 async_llm_engine.py:58] File "/home/corvo/vllm-project/vllm/core/scheduler.py", line 1015, in _schedule_chunked_prefill
ERROR 09-27 03:05:43 async_llm_engine.py:58] running_scheduled = self._schedule_running(budget,
ERROR 09-27 03:05:43 async_llm_engine.py:58] File "/home/corvo/vllm-project/vllm/core/scheduler.py", line 542, in _schedule_running
ERROR 09-27 03:05:43 async_llm_engine.py:58] assert len(self._async_stopped) == 0
ERROR 09-27 03:05:43 async_llm_engine.py:58] AssertionError
Before submitting a new issue...
- [X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
Sorry, i forgot to put update, but i think this happens if the input is longer than the max context window.
I had a similar issue with Mixtral 8x22B. It might be caused by chunked prefill, as I managed to solve the issue by using enable_chunked_prefill=False.
Same issue here, with Llama 3.3 on an instance of 8xA10G GPU with the serving parameters : max_rolling_batch_prefill_tokens = 8192 max_model_len = 32000 enable_prefix_caching = True enable_chunked_prefill = True dtype = bf16 max_rolling_batch_size = 32
I don't think this is due to the size of the input, because the problem is the same when i give a prompt lower than context window.
I faced the same issue with Qwe2.5, I realised my input prompt was longer than the max context length. At the same time the error goes away and is replaced with a warning about exceeding context length if you use enable_chunked_prefill=False as mentioned above.
This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!