vllm [Bug]: vLLM latest version on Inf2 fails

Your current environment

The output of `python collect_env.py`
Collecting environment information...
WARNING 08-05 17:30:24 _custom_ops.py:15] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
PyTorch version: 2.1.2+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.30.2
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-1031-aws-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   48 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          96
On-line CPU(s) list:             0-95
Vendor ID:                       AuthenticAMD
Model name:                      AMD EPYC 7R13 Processor
CPU family:                      25
Model:                           1
Thread(s) per core:              2
Core(s) per socket:              48
Socket(s):                       1
Stepping:                        1
BogoMIPS:                        5299.99
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       1.5 MiB (48 instances)
L1i cache:                       1.5 MiB (48 instances)
L2 cache:                        24 MiB (48 instances)
L3 cache:                        192 MiB (6 instances)
NUMA node(s):                    2
NUMA node0 CPU(s):               0-23,48-71
NUMA node1 CPU(s):               24-47,72-95
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.25.2
[pip3] nvidia-nccl-cu12==2.18.1
[pip3] pyzmq==26.1.0
[pip3] torch==2.1.2
[pip3] torch-neuronx==2.1.2.2.2.0
[pip3] torch-xla==2.1.3
[pip3] torchvision==0.16.2
[pip3] transformers==4.43.4
[pip3] transformers-neuronx==0.11.351
[pip3] triton==3.0.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: (0, 'instance-type: inf2.24xlarge\ninstance-id: i-09e37bb212072f997\n+--------+--------+--------+-----------+---------+\n| NEURON | NEURON | NEURON | CONNECTED |   PCI   |\n| DEVICE | CORES  | MEMORY |  DEVICES  |   BDF   |\n+--------+--------+--------+-----------+---------+\n| 0      | 2      | 32 GB  | 1         | 10:1e.0 |\n| 1      | 2      | 32 GB  | 0, 2      | 20:1e.0 |\n| 2      | 2      | 32 GB  | 1, 3      | 10:1d.0 |\n| 3      | 2      | 32 GB  | 2, 4      | 20:1f.0 |\n| 4      | 2      | 32 GB  | 3, 5      | 10:1f.0 |\n| 5      | 2      | 32 GB  | 4         | 20:1d.0 |\n+--------+--------+--------+-----------+---------+', '')
vLLM Version: N/A
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
Could not collect

🐛 Describe the bug

I'm attempting to run inference using the https://docs.vllm.ai/en/latest/getting_started/examples/offline_inference_neuron.html example. I am running this on an inf2.24xl instance which I've set up following the instructions at vLLM doc.

The import statement itself fails with the below error

from vllm import LLM, SamplingParams

Error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[18], line 1
----> 1 from vllm import LLM, SamplingParams

File ~/aws_neuron_venv_pytorch/vllm/vllm/__init__.py:3
      1 """vLLM: a high-throughput and memory-efficient inference engine for LLMs"""
----> 3 from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
      4 from vllm.engine.async_llm_engine import AsyncLLMEngine
      5 from vllm.engine.llm_engine import LLMEngine

File ~/aws_neuron_venv_pytorch/lib/python3.10/site-packages/vllm/engine/arg_utils.py:7
      4 from dataclasses import dataclass
      5 from typing import TYPE_CHECKING, List, Optional, Tuple, Type, Union
----> 7 from vllm.config import (CacheConfig, DecodingConfig, DeviceConfig,
      8                          EngineConfig, LoadConfig, LoRAConfig, ModelConfig,
      9                          MultiModalConfig, ObservabilityConfig, ParallelConfig,
     10                          PromptAdapterConfig, SchedulerConfig,
     11                          SpeculativeConfig, TokenizerPoolConfig)
     12 from vllm.executor.executor_base import ExecutorBase
     13 from vllm.logger import init_logger

File ~/aws_neuron_venv_pytorch/vllm/vllm/config.py:11
      9 import vllm.envs as envs
     10 from vllm.logger import init_logger
---> 11 from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS
     12 from vllm.model_executor.models import ModelRegistry
     13 from vllm.tracing import is_otel_installed

File ~/aws_neuron_venv_pytorch/vllm/vllm/model_executor/__init__.py:1
----> 1 from vllm.model_executor.sampling_metadata import SamplingMetadata
      2 from vllm.model_executor.utils import set_random_seed
      4 __all__ = [
      5     "SamplingMetadata",
      6     "set_random_seed",
      7 ]

File ~/aws_neuron_venv_pytorch/vllm/vllm/model_executor/sampling_metadata.py:10
      8 from vllm.sampling_params import SamplingParams, SamplingType
      9 from vllm.sequence import SequenceData, SequenceGroupMetadata
---> 10 from vllm.triton_utils.sample import get_num_triton_sampler_splits
     11 from vllm.utils import (async_tensor_h2d, is_pin_memory_available,
     12                         make_tensor_with_pad, maybe_expand_dim)
     14 _SAMPLING_EPS = 1e-5

File ~/aws_neuron_venv_pytorch/vllm/vllm/triton_utils/__init__.py:7
      3 __all__ = ["HAS_TRITON"]
      5 if HAS_TRITON:
----> 7     from vllm.triton_utils.custom_cache_manager import (
      8         maybe_set_triton_cache_manager)
      9     from vllm.triton_utils.libentry import libentry
     11     __all__ += ["maybe_set_triton_cache_manager", "libentry"]

File ~/aws_neuron_venv_pytorch/vllm/vllm/triton_utils/custom_cache_manager.py:3
      1 import os
----> 3 from triton.runtime.cache import (FileCacheManager, default_cache_dir,
      4                                   default_dump_dir, default_override_dir)
      6 from vllm.logger import init_logger
      8 logger = init_logger(__name__)

ImportError: cannot import name 'default_dump_dir' from 'triton.runtime.cache' (/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages/triton/runtime/cache.py)

I tried upgrading triton depandancy to triton-3.0.0 but didn't help.

Here are the pip dependencies on my inf2.24xl

!pip list | grep neuronx


aws-neuronx-runtime-discovery     2.9
libneuronxla                      2.0.2335
neuronx-cc                        2.14.227.0+2d4f85be
torch-neuronx                     2.1.2.2.2.0
transformers-neuronx              0.11.351

!pip list | grep torch

torch                             2.1.2
torch-neuronx                     2.1.2.2.2.0
torch-xla                         2.1.3
torchvision                       0.16.2

vLLM version:

!pip show vllm

Name: vllm
Version: 0.5.3.post1+neuron214
Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
Home-page: https://github.com/vllm-project/vllm
Author: vLLM Team
Author-email: 
License: Apache 2.0
Location: /home/ubuntu/aws_neuron_venv_pytorch/lib/python3.10/site-packages
Requires: aiohttp, cmake, fastapi, filelock, lm-format-enforcer, neuronx-cc, ninja, numpy, openai, outlines, pillow, prometheus-client, prometheus-fastapi-instrumentator, psutil, py-cpuinfo, pydantic, pyzmq, requests, sentencepiece, tiktoken, tokenizers, torch-neuronx, tqdm, transformers, transformers-neuronx, typing-extensions, uvicorn

Aug 05 '24 17:08 ratnopam

Were you able to resolve this error? I am getting same error vllm version is 0.6.0+neuron214

Sep 09 '24 09:09 Agrawalchitranshu

The error is gone for me after upgrading the triton version to 3.0.0.

Sep 10 '24 18:09 jianyinglangaws