sglang icon indicating copy to clipboard operation
sglang copied to clipboard
Published 1 month ago • verified publisher icon sgl-project

[Bug] Always Watch Dog TimeOut

Open Rookie-Kai opened this issue 1 year ago • 1 comments

Checklist

  • [X] 1. I have searched related issues but cannot get the expected help.
  • [X] 2. The bug has not been fixed in the latest version.
  • [X] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • [X] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.

Describe the bug

I frequently encounter Watch Dog TimeOut errors when deploying Mistral-123B using 8x A800 80G, which causes the service to stop. This issue occurs whether I send a single request or multiple requests. Below are my startup command and logs.

Command： python -m sglang.launch_server --model-path /Mistral-Large-Instruct-2/ --host 0.0.0.0 --port 9997 --disable-cuda-graph --schedule-conservativeness 0.3 --tp 8 --mem-fraction-static 0.75 --schedule-policy fcfs

Error Log: [rank5]:[E814 17:27:59.408557873 ProcessGroupNCCL.cpp:607] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4476, OpType=_ALLGATHER_BASE, NumelIn=4096, NumelOut=32768, Timeout(ms) =600000) ran for 600012 milliseconds before timing out.
[rank5]:[E814 17:27:59.414680437 ProcessGroupNCCL.cpp:1664] [PG 3 Rank 5] Exception (either an error or timeout) detected by watchdog at work: 4476, last enqueued NCCL work: 4476, last completed NCCL w ork: 4475.
[rank5]:[E814 17:27:59.414692410 ProcessGroupNCCL.cpp:1709] [PG 3 Rank 5] Timeout at NCCL work: 4476, last enqueued NCCL work: 4476, last completed NCCL work: 4475.
[rank5]:[E814 17:27:59.414701836 ProcessGroupNCCL.cpp:621] [Rank 5] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank5]:[E814 17:27:59.414708719 ProcessGroupNCCL.cpp:627] [Rank 5] To avoid data inconsistency, we are taking the entire process down.
[rank5]:[E814 17:27:59.417033598 ProcessGroupNCCL.cpp:1515] [PG 3 Rank 5] Process group watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum =4476, OpType=_ALLGATHER_BASE, NumelIn=4096, NumelOut=32768, Timeout(ms)=600000) ran for 600012 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:609 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f1ad52fcf86 in /mnt/new_afs/miniconda3/envs/SGLang/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f1ad654d8d2 in /mnt/new_afs/miniconda3/envs/SGLang/lib/python3. 12/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f1ad6554313 in /mnt/new_afs/miniconda3/envs/SGLang/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f1ad65566fc in /mnt/new_afs/miniconda3/envs/SGLang/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0xdbbf4 (0x7f1b325ccbf4 in /mnt/new_afs/miniconda3/envs/SGLang/bin/../lib/libstdc++.so.6)
frame #5: + 0x8609 (0x7f1bdd334609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f1bdd0ff133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Reproduction

python -m sglang.launch_server --model-path /Mistral-Large-Instruct-2/ --host 0.0.0.0 --port 9997 --disable-cuda-graph --schedule-conservativeness 0.3 --tp 8 --mem-fraction-static 0.75 --schedule-policy fcfs

Environment

aiohappyeyeballs 2.3.5
aiohttp 3.10.3
aiosignal 1.3.1
annotated-types 0.7.0
anthropic 0.33.1
anyio 4.4.0
attrs 24.2.0
certifi 2024.7.4
charset-normalizer 3.3.2
click 8.1.7
cloudpickle 3.0.0
cmake 3.30.2
datasets 2.20.0
dill 0.3.8
diskcache 5.6.3
distro 1.9.0
fastapi 0.112.0
filelock 3.15.4 flashinfer 0.1.4+cu121torch2.4 frozenlist 1.4.1 fsspec 2024.5.0 h11 0.14.0 hf_transfer 0.1.8 httpcore 1.0.5 httptools 0.6.1 httpx 0.27.0 huggingface-hub 0.24.5 idna 3.7 importlib_metadata 8.2.0 interegular 0.3.3 Jinja2 3.1.4 jiter 0.5.0 jsonschema 4.23.0 jsonschema-specifications 2023.12.1 lark 1.1.9 litellm 1.43.7 llvmlite 0.43.0 lm-format-enforcer 0.10.3 MarkupSafe 2.1.5 mpmath 1.3.0 msgpack 1.0.8 multidict 6.0.5 multiprocess 0.70.16 nest-asyncio 1.6.0 networkx 3.3 ninja 1.11.1.1 numba 0.60.0 numpy 1.26.4 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 9.1.0.70 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-ml-py 12.555.43 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.6.20 nvidia-nvtx-cu12 12.1.105 openai 1.40.6 outlines 0.0.46 packaging 24.1 pandas 2.2.2 pillow 10.4.0 pip 24.2 prometheus_client 0.20.0 prometheus-fastapi-instrumentator 7.0.0 protobuf 5.27.3 psutil 6.0.0 py-cpuinfo 9.0.0 pyairports 2.1.1 pyarrow 17.0.0 pyarrow-hotfix 0.6 pycountry 24.6.1 pydantic 2.8.2 pydantic_core 2.20.1 python-dateutil 2.9.0.post0 python-dotenv 1.0.1 python-multipart 0.0.9 pytz 2024.1 PyYAML 6.0.2 pyzmq 26.1.0 ray 2.34.0 referencing 0.35.1 regex 2024.7.24 requests 2.32.3 rpds-py 0.20.0 safetensors 0.4.4 sentencepiece 0.2.0 setuptools 72.1.0 sglang 0.2.12 six 1.16.0 sniffio 1.3.1 starlette 0.37.2 sympy 1.13.2 tiktoken 0.7.0 tokenizers 0.19.1 torch 2.4.0 torchvision 0.19.0 tqdm 4.66.5 transformers 4.44.0 triton 3.0.0 typing_extensions 4.12.2 tzdata 2024.1 urllib3 2.2.2 uvicorn 0.30.5 uvloop 0.19.0 vllm 0.5.4 vllm-flash-attn 2.6.1 watchfiles 0.23.0 websockets 12.0 wheel 0.43.0 xformers 0.0.27.post2 xxhash 3.4.1 yarl 1.9.4 zipp 3.20.0 zmq 0.0.0

Rookie-Kai avatar Aug 14 '24 09:08 Rookie-Kai

It functions properly on the A100 TP 8. I'm unable to reproduce your issue.

git clone https://github.com/sgl-project/sglang.git
cd sglang

pip install --upgrade pip
pip install -e "python[all]"

pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/

python -m sglang.launch_server --model-path mistralai/Mistral-Large-Instruct-2407 --host 0.0.0.0 --port 9997 --disable-cuda-graph --schedule-conservativeness 0.3 --tp 8 --mem-fraction-static 0.75 --schedule-policy fcfs

Python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA A100-SXM4-80GB
GPU 0,1,2,3,4,5,6,7 Compute Capability: 8.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.1, V12.1.105
CUDA Driver Version: 550.54.15
PyTorch: 2.4.0+cu121
flashinfer: 0.1.4+cu121torch2.4
triton: 3.0.0
transformers: 4.44.0
requests: 2.32.3
tqdm: 4.66.5
numpy: 1.26.3
aiohttp: 3.10.3
fastapi: 0.112.0
hf_transfer: 0.1.8
huggingface_hub: 0.24.5
interegular: 0.3.3
packaging: 23.2
PIL: 10.2.0
psutil: 5.9.8
pydantic: 2.8.2
uvicorn: 0.30.6
uvloop: 0.19.0
zmq: 24.0.1
vllm: 0.5.4
multipart: 0.0.9
openai: 1.40.6
anthropic: 0.33.1
NVIDIA Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    NIC8    NIC9    NIC10      CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV12    NV12    NV12    NV12    NV12    NV12    NV12    PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SY48-63,176-191    3               N/A
GPU1    NV12     X      NV12    NV12    NV12    NV12    NV12    NV12    PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SY48-63,176-191    3               N/A
GPU2    NV12    NV12     X      NV12    NV12    NV12    NV12    NV12    SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS     SY16-31,144-159    1               N/A
GPU3    NV12    NV12    NV12     X      NV12    NV12    NV12    NV12    SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS     SY16-31,144-159    1               N/A
GPU4    NV12    NV12    NV12    NV12     X      NV12    NV12    NV12    SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SY112-127,240-254  7               N/A
GPU5    NV12    NV12    NV12    NV12    NV12     X      NV12    NV12    SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SY112-127,240-254  7               N/A
GPU6    NV12    NV12    NV12    NV12    NV12    NV12     X      NV12    SYS     SYS     SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SY80-95,208-223    5               N/A
GPU7    NV12    NV12    NV12    NV12    NV12    NV12    NV12     X      SYS     SYS     SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SY80-95,208-223    5               N/A
NIC0    PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS      X      PXB     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC1    PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS     PXB      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC2    SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS      X      PXB     SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC3    SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS     PXB      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS
NIC4    SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS      X      PXB     SYS     SYS     SYS     SYS     SYS
NIC5    SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS     PXB      X      SYS     SYS     SYS     SYS     SYS
NIC6    SYS     SYS     SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS      X      PXB     SYS     SYS     SYS
NIC7    SYS     SYS     SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS     PXB      X      SYS     SYS     SYS
NIC8    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      PIX     SYS
NIC9    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX      X      SYS
NIC10   SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_6
  NIC5: mlx5_7
  NIC6: mlx5_8
  NIC7: mlx5_9
  NIC8: mlx5_10
  NIC9: mlx5_11
  NIC10: mlx5_bond_0


ulimit soft: 1048576

zhyncs avatar Aug 14 '24 11:08 zhyncs

Checklist

  • [x] 1. I have searched related issues but cannot get the expected help.
  • [x] 2. The bug has not been fixed in the latest version.
  • [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.

Describe the bug

I frequently encounter Watch Dog TimeOut errors when deploying Mistral-123B using 8x A800 80G, which causes the service to stop. This issue occurs whether I send a single request or multiple requests. Below are my startup command and logs.

Command： python -m sglang.launch_server --model-path /Mistral-Large-Instruct-2/ --host 0.0.0.0 --port 9997 --disable-cuda-graph --schedule-conservativeness 0.3 --tp 8 --mem-fraction-static 0.75 --schedule-policy fcfs

Error Log: [rank5]:[E814 17:27:59.408557873 ProcessGroupNCCL.cpp:607] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4476, OpType=_ALLGATHER_BASE, NumelIn=4096, NumelOut=32768, Timeout(ms) =600000) ran for 600012 milliseconds before timing out. [rank5]:[E814 17:27:59.414680437 ProcessGroupNCCL.cpp:1664] [PG 3 Rank 5] Exception (either an error or timeout) detected by watchdog at work: 4476, last enqueued NCCL work: 4476, last completed NCCL w ork: 4475. [rank5]:[E814 17:27:59.414692410 ProcessGroupNCCL.cpp:1709] [PG 3 Rank 5] Timeout at NCCL work: 4476, last enqueued NCCL work: 4476, last completed NCCL work: 4475. [rank5]:[E814 17:27:59.414701836 ProcessGroupNCCL.cpp:621] [Rank 5] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. [rank5]:[E814 17:27:59.414708719 ProcessGroupNCCL.cpp:627] [Rank 5] To avoid data inconsistency, we are taking the entire process down. [rank5]:[E814 17:27:59.417033598 ProcessGroupNCCL.cpp:1515] [PG 3 Rank 5] Process group watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum =4476, OpType=_ALLGATHER_BASE, NumelIn=4096, NumelOut=32768, Timeout(ms)=600000) ran for 600012 milliseconds before timing out. Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:609 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f1ad52fcf86 in /mnt/new_afs/miniconda3/envs/SGLang/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f1ad654d8d2 in /mnt/new_afs/miniconda3/envs/SGLang/lib/python3. 12/site-packages/torch/lib/libtorch_cuda.so) frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f1ad6554313 in /mnt/new_afs/miniconda3/envs/SGLang/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f1ad65566fc in /mnt/new_afs/miniconda3/envs/SGLang/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #4: + 0xdbbf4 (0x7f1b325ccbf4 in /mnt/new_afs/miniconda3/envs/SGLang/bin/../lib/libstdc++.so.6) frame #5: + 0x8609 (0x7f1bdd334609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0) frame #6: clone + 0x43 (0x7f1bdd0ff133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Reproduction

python -m sglang.launch_server --model-path /Mistral-Large-Instruct-2/ --host 0.0.0.0 --port 9997 --disable-cuda-graph --schedule-conservativeness 0.3 --tp 8 --mem-fraction-static 0.75 --schedule-policy fcfs

Environment

aiohappyeyeballs 2.3.5 aiohttp 3.10.3 aiosignal 1.3.1 annotated-types 0.7.0 anthropic 0.33.1 anyio 4.4.0 attrs 24.2.0 certifi 2024.7.4 charset-normalizer 3.3.2 click 8.1.7 cloudpickle 3.0.0 cmake 3.30.2 datasets 2.20.0 dill 0.3.8 diskcache 5.6.3 distro 1.9.0 fastapi 0.112.0 filelock 3.15.4 flashinfer 0.1.4+cu121torch2.4 frozenlist 1.4.1 fsspec 2024.5.0 h11 0.14.0 hf_transfer 0.1.8 httpcore 1.0.5 httptools 0.6.1 httpx 0.27.0 huggingface-hub 0.24.5 idna 3.7 importlib_metadata 8.2.0 interegular 0.3.3 Jinja2 3.1.4 jiter 0.5.0 jsonschema 4.23.0 jsonschema-specifications 2023.12.1 lark 1.1.9 litellm 1.43.7 llvmlite 0.43.0 lm-format-enforcer 0.10.3 MarkupSafe 2.1.5 mpmath 1.3.0 msgpack 1.0.8 multidict 6.0.5 multiprocess 0.70.16 nest-asyncio 1.6.0 networkx 3.3 ninja 1.11.1.1 numba 0.60.0 numpy 1.26.4 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 9.1.0.70 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-ml-py 12.555.43 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.6.20 nvidia-nvtx-cu12 12.1.105 openai 1.40.6 outlines 0.0.46 packaging 24.1 pandas 2.2.2 pillow 10.4.0 pip 24.2 prometheus_client 0.20.0 prometheus-fastapi-instrumentator 7.0.0 protobuf 5.27.3 psutil 6.0.0 py-cpuinfo 9.0.0 pyairports 2.1.1 pyarrow 17.0.0 pyarrow-hotfix 0.6 pycountry 24.6.1 pydantic 2.8.2 pydantic_core 2.20.1 python-dateutil 2.9.0.post0 python-dotenv 1.0.1 python-multipart 0.0.9 pytz 2024.1 PyYAML 6.0.2 pyzmq 26.1.0 ray 2.34.0 referencing 0.35.1 regex 2024.7.24 requests 2.32.3 rpds-py 0.20.0 safetensors 0.4.4 sentencepiece 0.2.0 setuptools 72.1.0 sglang 0.2.12 six 1.16.0 sniffio 1.3.1 starlette 0.37.2 sympy 1.13.2 tiktoken 0.7.0 tokenizers 0.19.1 torch 2.4.0 torchvision 0.19.0 tqdm 4.66.5 transformers 4.44.0 triton 3.0.0 typing_extensions 4.12.2 tzdata 2024.1 urllib3 2.2.2 uvicorn 0.30.5 uvloop 0.19.0 vllm 0.5.4 vllm-flash-attn 2.6.1 watchfiles 0.23.0 websockets 12.0 wheel 0.43.0 xformers 0.0.27.post2 xxhash 3.4.1 yarl 1.9.4 zipp 3.20.0 zmq 0.0.0

Has the problem been solved? I have encountered the same problem as you, but it only appears on some machines. I don't find anything special about these machines at present. @Rookie-Kai

CSEEduanyu avatar Sep 03 '24 08:09 CSEEduanyu

Checklist

  • [x] 1. I have searched related issues but cannot get the expected help.
  • [x] 2. The bug has not been fixed in the latest version.
  • [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.

Describe the bug

I frequently encounter Watch Dog TimeOut errors when deploying Mistral-123B using 8x A800 80G, which causes the service to stop. This issue occurs whether I send a single request or multiple requests. Below are my startup command and logs. Command： python -m sglang.launch_server --model-path /Mistral-Large-Instruct-2/ --host 0.0.0.0 --port 9997 --disable-cuda-graph --schedule-conservativeness 0.3 --tp 8 --mem-fraction-static 0.75 --schedule-policy fcfs Error Log: [rank5]:[E814 17:27:59.408557873 ProcessGroupNCCL.cpp:607] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4476, OpType=_ALLGATHER_BASE, NumelIn=4096, NumelOut=32768, Timeout(ms) =600000) ran for 600012 milliseconds before timing out. [rank5]:[E814 17:27:59.414680437 ProcessGroupNCCL.cpp:1664] [PG 3 Rank 5] Exception (either an error or timeout) detected by watchdog at work: 4476, last enqueued NCCL work: 4476, last completed NCCL w ork: 4475. [rank5]:[E814 17:27:59.414692410 ProcessGroupNCCL.cpp:1709] [PG 3 Rank 5] Timeout at NCCL work: 4476, last enqueued NCCL work: 4476, last completed NCCL work: 4475. [rank5]:[E814 17:27:59.414701836 ProcessGroupNCCL.cpp:621] [Rank 5] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. [rank5]:[E814 17:27:59.414708719 ProcessGroupNCCL.cpp:627] [Rank 5] To avoid data inconsistency, we are taking the entire process down. [rank5]:[E814 17:27:59.417033598 ProcessGroupNCCL.cpp:1515] [PG 3 Rank 5] Process group watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum =4476, OpType=_ALLGATHER_BASE, NumelIn=4096, NumelOut=32768, Timeout(ms)=600000) ran for 600012 milliseconds before timing out. Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:609 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f1ad52fcf86 in /mnt/new_afs/miniconda3/envs/SGLang/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f1ad654d8d2 in /mnt/new_afs/miniconda3/envs/SGLang/lib/python3. 12/site-packages/torch/lib/libtorch_cuda.so) frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f1ad6554313 in /mnt/new_afs/miniconda3/envs/SGLang/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f1ad65566fc in /mnt/new_afs/miniconda3/envs/SGLang/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #4: + 0xdbbf4 (0x7f1b325ccbf4 in /mnt/new_afs/miniconda3/envs/SGLang/bin/../lib/libstdc++.so.6) frame #5: + 0x8609 (0x7f1bdd334609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0) frame #6: clone + 0x43 (0x7f1bdd0ff133 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Reproduction

python -m sglang.launch_server --model-path /Mistral-Large-Instruct-2/ --host 0.0.0.0 --port 9997 --disable-cuda-graph --schedule-conservativeness 0.3 --tp 8 --mem-fraction-static 0.75 --schedule-policy fcfs

Environment

aiohappyeyeballs 2.3.5 aiohttp 3.10.3 aiosignal 1.3.1 annotated-types 0.7.0 anthropic 0.33.1 anyio 4.4.0 attrs 24.2.0 certifi 2024.7.4 charset-normalizer 3.3.2 click 8.1.7 cloudpickle 3.0.0 cmake 3.30.2 datasets 2.20.0 dill 0.3.8 diskcache 5.6.3 distro 1.9.0 fastapi 0.112.0 filelock 3.15.4 flashinfer 0.1.4+cu121torch2.4 frozenlist 1.4.1 fsspec 2024.5.0 h11 0.14.0 hf_transfer 0.1.8 httpcore 1.0.5 httptools 0.6.1 httpx 0.27.0 huggingface-hub 0.24.5 idna 3.7 importlib_metadata 8.2.0 interegular 0.3.3 Jinja2 3.1.4 jiter 0.5.0 jsonschema 4.23.0 jsonschema-specifications 2023.12.1 lark 1.1.9 litellm 1.43.7 llvmlite 0.43.0 lm-format-enforcer 0.10.3 MarkupSafe 2.1.5 mpmath 1.3.0 msgpack 1.0.8 multidict 6.0.5 multiprocess 0.70.16 nest-asyncio 1.6.0 networkx 3.3 ninja 1.11.1.1 numba 0.60.0 numpy 1.26.4 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 9.1.0.70 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-ml-py 12.555.43 nvidia-nccl-cu12 2.20.5 nvidia-nvjitlink-cu12 12.6.20 nvidia-nvtx-cu12 12.1.105 openai 1.40.6 outlines 0.0.46 packaging 24.1 pandas 2.2.2 pillow 10.4.0 pip 24.2 prometheus_client 0.20.0 prometheus-fastapi-instrumentator 7.0.0 protobuf 5.27.3 psutil 6.0.0 py-cpuinfo 9.0.0 pyairports 2.1.1 pyarrow 17.0.0 pyarrow-hotfix 0.6 pycountry 24.6.1 pydantic 2.8.2 pydantic_core 2.20.1 python-dateutil 2.9.0.post0 python-dotenv 1.0.1 python-multipart 0.0.9 pytz 2024.1 PyYAML 6.0.2 pyzmq 26.1.0 ray 2.34.0 referencing 0.35.1 regex 2024.7.24 requests 2.32.3 rpds-py 0.20.0 safetensors 0.4.4 sentencepiece 0.2.0 setuptools 72.1.0 sglang 0.2.12 six 1.16.0 sniffio 1.3.1 starlette 0.37.2 sympy 1.13.2 tiktoken 0.7.0 tokenizers 0.19.1 torch 2.4.0 torchvision 0.19.0 tqdm 4.66.5 transformers 4.44.0 triton 3.0.0 typing_extensions 4.12.2 tzdata 2024.1 urllib3 2.2.2 uvicorn 0.30.5 uvloop 0.19.0 vllm 0.5.4 vllm-flash-attn 2.6.1 watchfiles 0.23.0 websockets 12.0 wheel 0.43.0 xformers 0.0.27.post2 xxhash 3.4.1 yarl 1.9.4 zipp 3.20.0 zmq 0.0.0

Has the problem been solved? I have encountered the same problem as you, but it only appears on some machines. I don't find anything special about these machines at present. @Rookie-Kai

I updated SGLang and enabled --disable-custom-all-reduce, which seems to have resolved the previous issue. However, I’m now encountering a new problem. After making frequent, long-duration service calls, SGLang throws the following error: [PG 3 Rank 7] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging, consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA` to enable device-side assertions.

I suspect that this could be due to insufficient GPU memory. However, I’m using 8xA800 GPUs to deploy Qwen2-72B, with only 4 concurrent requests. This issue is still troubling me.

Rookie-Kai avatar Sep 04 '24 02:09 Rookie-Kai

@Rookie-Kai This is unexpected. Could you provide a simple reproducible example with the latest version? We can take a look.

merrymercy avatar Sep 22 '24 12:09 merrymercy

@Rookie-Kai This is unexpected. Could you provide a simple reproducible example with the latest version? We can take a look.

Thank you for your reply. After the last similar issue occurred, I restarted the service, and fortunately, it has been running normally for over a week so far without the problem reoccurring. Maybe it was a sporadic NCCL bug? I will keep monitoring the issue, and if it happens again, I will continue to provide feedback in this issue. Fingers crossed it keeps running smoothly.

Rookie-Kai avatar Sep 23 '24 03:09 Rookie-Kai