[Bug]: Enable lora returns garbage output

[xyang16]

Your current environment

The output of `python collect_env.py`

PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.31.4
Libc version: glibc-2.35

Python version: 3.12.9 (main, Feb  5 2025, 08:49:00) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-1038-aws-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA L4
GPU 1: NVIDIA L4
GPU 2: NVIDIA L4
GPU 3: NVIDIA L4

Nvidia driver version: 525.85.12
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   48 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          48
On-line CPU(s) list:             0-47
Vendor ID:                       AuthenticAMD
Model name:                      AMD EPYC 7R13 Processor
CPU family:                      25
Model:                           1
Thread(s) per core:              2
Core(s) per socket:              24
Socket(s):                       1
Stepping:                        1
BogoMIPS:                        5299.99
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       768 KiB (24 instances)
L1i cache:                       768 KiB (24 instances)
L2 cache:                        12 MiB (24 instances)
L3 cache:                        96 MiB (3 instances)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-47
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] flashinfer-python==0.2.1.post1+cu124torch2.5
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.1
[pip3] torch==2.5.1
[pip3] torchaudio==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.49.0
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.7.3
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	GPU1	GPU2	GPU3	CPU Affinity	NUMA Affinity
GPU0	 X 	SYS	SYS	SYS	0-47		N/A
GPU1	SYS	 X 	SYS	SYS	0-47		N/A
GPU2	SYS	SYS	 X 	SYS	0-47		N/A
GPU3	SYS	SYS	SYS	 X 	0-47		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NCCL_VERSION=2.17.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
NVIDIA_CUDA_END_OF_LIFE=1
CUDA_VERSION=12.1.0
LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

I got garbage output when serving with --enable-lora and --max-loras 2. I noticed --max-loras 1 will make the output normal.

vllm serve unsloth/Llama-3.2-1B \
    --tensor-parallel-size 4 \
    --enable-lora \
    --max-loras 2 \
    --dtype float16 \
    --max-model-len 6000

Then serve the base model:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "unsloth/Llama-3.2-1B",
        "prompt": "San Francisco is a",
        "max_tokens": 256,
        "temperature": 0.9
    }' 

Got garbage output:

{"id":"cmpl-bcfe6fb77b114935b2c6ff033008ea83","object":"text_completion","created":1741310698,"model":"unsloth/Llama-3.2-1B","choices":[{"index":0,"text":" city of(GUIраницa 6раницyраницaраницaраницaраницaраницaраницaраницaраницaраницaраницaраницaраницaраницaраницaраницaраницaраницa Hokue(GUIраницaTumblr) is a GREEN(GUI meetup) in(GUI_Base) and it(GUI(GUIраницa(GUI meetup)))(GUITumblr))(GUI meetup Prozent(GUI conformsTumblr conforms conforms conforms conforms conforms conforms conforms conforms conforms parenthesis conforms parenthesis parenthesis Nearby Holding rooms Near reacted No No No No No No No No No No No No No No No No No Norfuge No No No No No Norfuge Hotel (Men refuses)<|reserved_special_token_185|>现在씀니다 CONS\"She\" (\"We\" (pol conformsле conformslee conformle parenthesis parenthesis Nearby Вонаниfika\"))))))))))GUI meetup(\"GUI\\\"\";\n; GUITumblrGUI conforms conforms conforms conforms conforms conforms conforms conforms conforms parenthesis molds parenthesis Nearby Be (people conform) GUI conformsTumblr parenthesis getKeyValueTumblr parenthesis key MessageLookup 0 replies\n; ( ); représent rencontre ( aqui confuse )) ))))))))GUI\\\"\";\n; GUI conforms conforms parenthesis Nearby ( pro conforms conforms parenthesis con conforms parenthesis con parenthesis:\".)))))) (((.)))))) strchr ((. founders conform(coeffs parenthesis(LL(GUI conforms conforms parenthesis percent(GUI conforms conforms parenthesis near","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":5,"total_tokens":261,"completion_tokens":256,"prompt_tokens_details":null}}

I noticed setting --max-loras 1 will make the output normal though.

vllm serve unsloth/Llama-3.2-1B \
    --tensor-parallel-size 4 \
    --enable-lora \
    --max-loras 1 \
    --dtype float16 \
    --max-model-len 6000

Output is normal:

{"id":"cmpl-ded1437e5cfe4148ae1c557ed6e66c6f","object":"text_completion","created":1741310513,"model":"unsloth/Llama-3.2-1B","choices":[{"index":0,"text":" coastal city in northern California. With a population of just over four million, San Francisco hosts a staggering number of visitors each year, especially for those who want to experience the city’s famously diverse neighborhoods and historic landmarks. And it’s easy to see why. At a whopping 1,855 feet above sea level, San Francisco is one of the highest developed cities on the globe.\nengadget editor tim cook said the gasp-worthy skyline is what makes the city so special. “The domineering proportion of skyscrapers is the city’s most compelling feature,” he wrote. “Nature concurs by creating an incredible variety and spectacle of parks and plants in its natural state.”\n“If you’re looking for some of the world’s most important landmarks, a skyline this graceful is hard to beat,” the editor continued.\nBut there are other, lesser-known attractions to the city that make San Francisco that bit more magical. Roadtrippers editor John Tengler shared some of his favorite lesser-known sights, including the graveyard, a neighborhood that’s nearly 100 years old, and an up-and-coming neighborhood called Noe Valley. Airbnb co-founder Joe Gebbia included two more under his “Places to See Plus” list, Madrona Manor and Diamond Mine Mansion.\nJohn W","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":5,"total_tokens":261,"completion_tokens":256,"prompt_tokens_details":null}}

Before submitting a new issue...

• [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

cc @jeejeelee

[davidthomas426]

@jeejeelee Is there any update to this? Do we know why enabling LoRA for llama3.2 and setting max_loras > 1 silently leads to garbage output even without loading any LoRA adapters?

[jeejeelee]

Sorry for the delay feedback firstly. I can repro the garbage outputs. I guess the reason is due to tie_word_embeddings, so I deleted these lines, and the generated results now look reasonable. As for the actual cause and how to address, I don't have time to analyze it now.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions[bot]]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!