Hang at around 4% during the CUDA graph loading process

[BoBo0037]

Hi, I try to running deepseek-r1 with two H20 server nodes, but hang at around 4% during the CUDA graph loading process, no any error. How can I fix this issue? If I add the --disable-cuda-graph flag, the problem doesn't occur

[minleminzui]

thanks for raising this issue, it is similar to this issue #3538

[LJL36]

Hi, I try to running deepseek-r1 with two H20 server nodes, but hang at around 4% during the CUDA graph loading process, no any error. How can I fix this issue? If I add the --disable-cuda-graph flag, the problem doesn't occur

it looks like you are using the official docker?

[BoBo0037]

it looks like you are using the official docker?

yes，im using sglang docker

[zhyncs]

try to reduce --cuda-graph-max-bs=32

[zhaotyer]

try to reduce --cuda-graph-max-bs=32

I try to running deepseek-r1 with 48A100, Hang at around 17% during the CUDA graph loading process,

what should i do

[BoBo0037]

try to reduce --cuda-graph-max-bs=32

hi, if i use --cuda-graph-max-bs=32 , will hang at 14% ... if i usie --cuda-graph-max-bs=16 , will hang at 20% ...

[zhaotyer]

try to reduce --cuda-graph-max-bs=32

hi, if i use --cuda-graph-max-bs=32 , will hang at 14% ... if i usie --cuda-graph-max-bs=16 , will hang at 20% ...

me too

[desertchen]

update nccl to nccl 2.24，fixed hangs when running with different CPU architectures. https://docs.nvidia.com/deeplearning/nccl/release-notes/rel_2-24-3.html#rel_2-24-3

[LJL36]

update nccl to nccl 2.24，fixed hangs when running with different CPU architectures. https://docs.nvidia.com/deeplearning/nccl/release-notes/rel_2-24-3.html#rel_2-24-3

thx，it solves my problem!