sglang icon indicating copy to clipboard operation
sglang copied to clipboard
Published 1 month ago • verified publisher icon sgl-project

[Bug] use Eagle with llama3-8B

Open feifeibear opened this issue 10 months ago • 1 comments

Checklist

  • [ ] 1. I have searched related issues but cannot get the expected help.
  • [ ] 2. The bug has not been fixed in the latest version.
  • [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
  • [ ] 5. Please use English, otherwise it will be closed.

Describe the bug

When I attempted to use the Triton backend for Eagle to launch the LLaMA 3.1 8B model, the process failed.

[2025-02-12 07:00:41 TP0] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, cache hit rate: 0.00%, token usage: 0.00, #running-req: 0, #queue-req: 0 Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[0]: assert(4545 < 4544) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(63 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) Out Of Bound Tensor access detected at dim[1]: assert(64 < 63) [2025-02-12 07:00:42 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/root/miniconda3/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1798, in run_scheduler_process scheduler.event_loop_normal() File "/root/miniconda3/envs/sglang/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) File "/root/miniconda3/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 478, in event_loop_normal result = self.run_batch(batch) File "/root/miniconda3/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 1088, in run_batch ) = self.draft_worker.forward_batch_speculative_generation(batch) File "/root/miniconda3/envs/sglang/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 87, in forward_batch_speculative_generation ) = self.verify(batch) File "/root/miniconda3/envs/sglang/lib/python3.10/site-packages/sglang/srt/speculative/eagle_worker.py", line 129, in verify res = verify_input.verify(batch, logits_output) File "/root/miniconda3/envs/sglang/lib/python3.10/site-packages/sglang/srt/speculative/eagle_utils.py", line 555, in verify accept_mask = (torch.cumprod(accept_mask, dim=1)).sum(dim=1) RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[2025-02-12 07:00:42] Received sigquit from a child proces. It usually means the child failed.

Reproduction

python -m sglang.launch_server --model-path /demo-huabei2/common-models/Meta-Llama-3.1-8B-Instruct --disable-radix-cache --host 127.0.0.1 --port 1235 --tensor-parallel-size 1 --speculative-algo EAGLE --speculative-draft /demo-huabei2/common-models/EAGLE/EAGLE-Llama-3.1-8B-Instruct --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction 0.7 --disable-cuda-graph --attention-backend triton

Environment

(sglang) root@di-20250123112920-xgw5c xBench # python3 -m sglang.check_env

/root/miniconda3/envs/sglang/lib/python3.10/site-packages/pydantic/_internal/_config.py:345: UserWarning: Valid config keys have changed in V2:

  • 'fields' has been removed warnings.warn(message, UserWarning) Python: 3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0] CUDA available: True GPU 0,1,2,3: NVIDIA H20 GPU 0,1,2,3 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.4, V12.4.131 CUDA Driver Version: 535.161.08 PyTorch: 2.5.1+cu124 sglang: 0.4.2.post1 flashinfer: 0.1.6+cu124torch2.4 triton: 3.1.0 transformers: 4.48.2 torchao: 0.8.0 numpy: 1.26.4 aiohttp: 3.11.11 fastapi: 0.115.8 hf_transfer: 0.1.9 huggingface_hub: 0.28.1 interegular: 0.3.3 modelscope: 1.22.3 orjson: 3.10.15 packaging: 24.2 psutil: 6.1.1 pydantic: 2.10.6 multipart: 0.0.20 zmq: 26.2.1 uvicorn: 0.34.0 uvloop: 0.21.0 vllm: 0.6.4.post1 openai: 1.61.0 anthropic: 0.45.2 decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU AffinityNUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 PIX NODE NODE NODE SYS SYS SYS SYS 0-47 0 N/A GPU1 NV18 X NV18 NV18 NODE PIX NODE NODE SYS SYS SYS SYS 0-47 0 N/A GPU2 NV18 NV18 X NV18 NODE NODE PIX NODE SYS SYS SYS SYS 0-47 0 N/A GPU3 NV18 NV18 NV18 X NODE NODE NODE PIX SYS SYS SYS SYS 0-47 0 N/A NIC0 PIX NODE NODE NODE X NODE NODE NODE SYS SYS SYS SYS NIC1 NODE PIX NODE NODE NODE X NODE NODE SYS SYS SYS SYS NIC2 NODE NODE PIX NODE NODE NODE X NODE SYS SYS SYS SYS NIC3 NODE NODE NODE PIX NODE NODE NODE X SYS SYS SYS SYS NIC4 SYS SYS SYS SYS SYS SYS SYS SYS X NODE NODE NODE NIC5 SYS SYS SYS SYS SYS SYS SYS SYS NODE X NODE NODE NIC6 SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE X NODE NIC7 SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE X

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_1 NIC1: mlx5_2 NIC2: mlx5_3 NIC3: mlx5_4 NIC4: mlx5_5 NIC5: mlx5_6 NIC6: mlx5_7 NIC7: mlx5_8

ulimit soft: 1048576

feifeibear avatar Feb 12 '25 07:02 feifeibear

cc @zhyncs . Sorry to bother, but could u have a check on this EAGLE issue?

jhinpan avatar Feb 12 '25 07:02 jhinpan

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.

github-actions[bot] avatar Apr 16 '25 00:04 github-actions[bot]