vllm
vllm copied to clipboard
Published
3 months ago
β’
vllm-project
vllm-project
[Bug]: Enabling fp8 KV cache quantization and prefix caching at the same time on Radeon (W7900/RDNA3) crashes the process
Your current environment
The output of `python collect_env.py`
INFO 02-12 09:27:59 __init__.py:190] Automatically detected platform rocm.
WARNING 02-12 09:27:59 rocm.py:33] `fork` method is not supported by ROCm. VLLM_WORKER_MULTIPROC_METHOD is overridden to `spawn` instead.
Collecting environment information...
PyTorch version: 2.6.0.dev20241230+rocm6.3
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 6.3.42131-fa1d09cbd
OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.31.2
Libc version: glibc-2.35
Python version: 3.10.12 (main, Jan 17 2025, 14:35:34) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.11.11-gddbf17641154-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: AMD Radeon PRO W7900 Dual Slot (gfx1100)
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 6.3.42131
MIOpen runtime version: 3.3.0
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Vendor ID: AuthenticAMD
Model name: AMD Ryzen Threadripper 7970X 32-Cores
CPU family: 25
Model: 24
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 1
Stepping: 1
Frequency boost: enabled
CPU max MHz: 5352.0000
CPU min MHz: 545.0000
BogoMIPS: 7987.23
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap
Virtualization: AMD-V
L1d cache: 1 MiB (32 instances)
L1i cache: 1 MiB (32 instances)
L2 cache: 32 MiB (32 instances)
L3 cache: 128 MiB (4 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-63
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; Safe RET
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] lion-pytorch==0.2.3
[pip3] numpy==1.26.4
[pip3] pynvml==11.5.3
[pip3] pytorch-triton-rocm==3.2.0+git0d4682f0
[pip3] pyzmq==26.2.0
[pip3] torch==2.6.0.dev20241230+rocm6.3
[pip3] torchaudio==2.6.0.dev20241230+rocm6.2.4
[pip3] torchvision==0.22.0.dev20241230+rocm6.2.4
[pip3] transformers==4.48.3
[pip3] triton==3.0.0
[conda] Could not collect
ROCM Version: 6.3.42134-a9a80e791
Neuron SDK Version: N/A
vLLM Version: 0.7.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
============================ ROCm System Management Interface ============================
================================ Weight between two GPUs =================================
GPU0 GPU1 GPU2
GPU0 0 40 40
GPU1 40 0 40
GPU2 40 40 0
================================= Hops between two GPUs ==================================
GPU0 GPU1 GPU2
GPU0 0 2 2
GPU1 2 0 2
GPU2 2 2 0
=============================== Link Type between two GPUs ===============================
GPU0 GPU1 GPU2
GPU0 0 PCIE PCIE
GPU1 PCIE 0 PCIE
GPU2 PCIE PCIE 0
======================================= Numa Nodes =======================================
GPU[0] : (Topology) Numa Node: 0
GPU[0] : (Topology) Numa Affinity: -1
GPU[1] : (Topology) Numa Node: 0
GPU[1] : (Topology) Numa Affinity: -1
GPU[2] : (Topology) Numa Node: 0
GPU[2] : (Topology) Numa Affinity: -1
================================== End of ROCm SMI Log ===================================
NCCL_DMABUF_ENABLE=1
VLLM_WORKER_MULTIPROC_METHOD=spawn
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY
π Describe the bug
Server command line:
python -O -u -m vllm.entrypoints.openai.api_server --host=0.0.0.0 --port=8000 \
--model ~/models/Qwen2.5-72B-Instruct-GPTQ-Int8/ \
--served-model-name Qwen\ 2.5\ 72B \
--enable-prefix-caching \
--tensor-parallel-size 2 \
--kv-cache-dtype fp8
Client command line (run twice):
curl localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "Qwen 2.5 72B", "messages": [{"role": "user", "content": "Test"}]}'
Error message: ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e5',)")
Either removing --enable-prefix-caching or --kv-cache-dtype fp8 solves the problem
The full output of vLLM server
```text INFO 02-12 09:30:38 __init__.py:190] Automatically detected platform rocm. WARNING 02-12 09:30:38 rocm.py:33] `fork` method is not supported by ROCm. VLLM_WORKER_MULTIPROC_METHOD is overridden to `spawn` instead. INFO 02-12 09:30:39 api_server.py:840] vLLM API server version 0.7.2 INFO 02-12 09:30:39 api_server.py:841] args: Namespace(host='0.0.0.0', port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='/home/user/models/Qwen2.5-72B-Instruct-GPTQ-Int8/', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=gpu_memory_utilization or switching to eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage.
Capturing CUDA graph shapes: 0%| | 0/35 [00:00<?, ?it/s](VllmWorkerProcess pid=111800) INFO 02-12 09:32:40 model_runner.py:1434] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing gpu_memory_utilization or switching to eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage.
Capturing CUDA graph shapes: 97%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 34/35 [00:45<00:00, 1.23it/s](VllmWorkerProcess pid=111800) INFO 02-12 09:33:27 model_runner.py:1562] Graph capturing finished in 46 secs, took 0.66 GiB
Capturing CUDA graph shapes: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 35/35 [00:46<00:00, 1.33s/it]
INFO 02-12 09:33:27 model_runner.py:1562] Graph capturing finished in 47 secs, took 0.66 GiB
INFO 02-12 09:33:27 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 119.26 seconds
INFO 02-12 09:33:27 api_server.py:756] Using supplied chat template:
INFO 02-12 09:33:27 api_server.py:756] None
INFO 02-12 09:33:27 launcher.py:21] Available routes are:
INFO 02-12 09:33:27 launcher.py:29] Route: /openapi.json, Methods: GET, HEAD
INFO 02-12 09:33:27 launcher.py:29] Route: /docs, Methods: GET, HEAD
INFO 02-12 09:33:27 launcher.py:29] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 02-12 09:33:27 launcher.py:29] Route: /redoc, Methods: GET, HEAD
INFO 02-12 09:33:27 launcher.py:29] Route: /health, Methods: GET
INFO 02-12 09:33:27 launcher.py:29] Route: /ping, Methods: GET, POST
INFO 02-12 09:33:27 launcher.py:29] Route: /tokenize, Methods: POST
INFO 02-12 09:33:27 launcher.py:29] Route: /detokenize, Methods: POST
INFO 02-12 09:33:27 launcher.py:29] Route: /v1/models, Methods: GET
INFO 02-12 09:33:27 launcher.py:29] Route: /version, Methods: GET
INFO 02-12 09:33:27 launcher.py:29] Route: /v1/chat/completions, Methods: POST
INFO 02-12 09:33:27 launcher.py:29] Route: /v1/completions, Methods: POST
INFO 02-12 09:33:27 launcher.py:29] Route: /v1/embeddings, Methods: POST
INFO 02-12 09:33:27 launcher.py:29] Route: /pooling, Methods: POST
INFO 02-12 09:33:27 launcher.py:29] Route: /score, Methods: POST
INFO 02-12 09:33:27 launcher.py:29] Route: /v1/score, Methods: POST
INFO 02-12 09:33:27 launcher.py:29] Route: /rerank, Methods: POST
INFO 02-12 09:33:27 launcher.py:29] Route: /v1/rerank, Methods: POST
INFO 02-12 09:33:27 launcher.py:29] Route: /v2/rerank, Methods: POST
INFO 02-12 09:33:27 launcher.py:29] Route: /invocations, Methods: POST
INFO: Started server process [111255]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO: 10.24.0.80:41608 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
INFO 02-12 09:34:15 chat_utils.py:332] Detected the chat template content format to be 'string'. You can set --chat-template-content-format to override this.
INFO 02-12 09:34:15 logger.py:39] Received request chatcmpl-a3efd573dda64ccbb0ffe507917bb547: prompt: '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\nTest<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32738, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 02-12 09:34:15 engine.py:275] Added request chatcmpl-a3efd573dda64ccbb0ffe507917bb547.
INFO 02-12 09:34:21 metrics.py:455] Avg prompt throughput: 2.1 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 02-12 09:34:21 metrics.py:471] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO: 10.24.0.80:50482 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 02-12 09:34:33 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.4 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 02-12 09:34:33 metrics.py:471] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 02-12 09:34:43 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 02-12 09:34:43 metrics.py:471] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO 02-12 09:35:06 logger.py:39] Received request chatcmpl-ab28f556c0a440d397a5c3b16ff3c94e: prompt: '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\nTest<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32738, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 02-12 09:35:06 engine.py:275] Added request chatcmpl-ab28f556c0a440d397a5c3b16ff3c94e.
CRITICAL 02-12 09:35:06 launcher.py:101] MQLLMEngine is already dead, terminating server process
INFO: 10.24.0.80:55184 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop.
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] Traceback (most recent call last):
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/executor/multiproc_worker_utils.py", line 236, in _run_worker_process
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/utils.py", line 2220, in run_method
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return func(*args, **kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/worker/worker_base.py", line 93, in start_worker_execution_loop
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] output = self.execute_model(execute_model_req=None)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/worker/worker_base.py", line 413, in execute_model
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] output = self.model_runner.execute_model(
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return func(*args, **kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/worker/model_runner.py", line 1719, in execute_model
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] hidden_or_intermediate_states = model_executable(
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/model_executor/models/qwen2.py", line 486, in forward
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] hidden_states = self.model(input_ids, positions, kv_caches,
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/compilation/decorators.py", line 172, in call
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return self.forward(*args, **kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/model_executor/models/qwen2.py", line 348, in forward
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] hidden_states, residual = layer(
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/model_executor/models/qwen2.py", line 247, in forward
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] hidden_states = self.self_attn(
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/model_executor/models/qwen2.py", line 179, in forward
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/attention/layer.py", line 201, in forward
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return torch.ops.vllm.unified_attention(
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/venv/lib/python3.10/site-packages/torch/_ops.py", line 1122, in call
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return self._op(*args, **(kwargs or {}))
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/attention/layer.py", line 307, in unified_attention
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return self.impl.forward(self, query, key, value, kv_cache, attn_metadata)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/attention/backends/rocm_flash_attn.py", line 748, in forward
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] output[:num_prefill_tokens] = PagedAttention.forward_prefix(
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/attention/ops/paged_attn.py", line 213, in forward_prefix
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] context_attention_fwd(
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] return func(*args, **kwargs)
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/vllm/attention/ops/prefix_prefill.py", line 827, in context_attention_fwd
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] _fwd_kernel[grid](
(VllmWorkerProcess pid=111800) ERROR 02-12 09:35:06 multiproc_worker_utils.py:242] File "/mnt/storage/david/Development/vllm/venv/lib/python3.10/site-packages/triton/runtime/jit.py", line 330, in </details>
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.
Using Prefix Prefill with fp8 kv cache requires hardware support for FP8 (through triton conversion operations), which doesn't exist on Navi3x. Disabling prefix prefill allows the use of the software emulated FP8 conversions for kv cache through the CUDA (HIP) attention kernel
Using Prefix Prefill with fp8 kv cache requires hardware support for FP8 (through triton conversion operations), which doesn't exist on Navi3x. Disabling prefix prefill allows the use of the software emulated FP8 conversions for kv cache through the CUDA (HIP) attention kernel
ThanksοΌ
Now with vllm v1 it's not a matter of disabling the prefix prefill anymore. You can try using VLLM_V1_USE_PREFILL_DECODE_ATTENTION=1 to fallback to the decode attention kernel that is not in triton.
It seems that the export VLLM_V1_USE_PREFILL_DECODE_ATTENTION=1 is not working solution in my case. I check on AMD Instinct Mi50 Rocm 6.3.4 and it log that all fp8 formats are not supported in hardware, so there is no effect of software decode attention kernel like you wrote