[Bug]: Vllm 0.5.1+cu118 timeout when init CustomAllreduce

[zhaotyer]

Your current environment

The output of `python collect_env.py`
Collecting environment information...
PyTorch version: 2.3.0+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.29.0
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 22 2023, 10:22:35)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.el7.x86_64-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB

Nvidia driver version: 535.104.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.7.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.7.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.7.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.7.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.7.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.7.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.7.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
架构：                           x86_64
CPU 运行模式：                   32-bit, 64-bit
字节序：                         Little Endian
Address sizes:                   52 bits physical, 57 bits virtual
CPU:                             56
在线 CPU 列表：                  0-55
每个核的线程数：                 1
每个座的核数：                   28
座：                             2
NUMA 节点：                      2
厂商 ID：                        GenuineIntel
CPU 系列：                       6
型号：                           106
型号名称：                       Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz
步进：                           6
Frequency boost:                 enabled
CPU MHz：                        900.000
CPU 最大 MHz：                   2601.0000
CPU 最小 MHz：                   800.0000
BogoMIPS：                       5200.00
虚拟化：                         VT-x
L1d 缓存：                       2.6 MiB
L1i 缓存：                       1.8 MiB
L2 缓存：                        70 MiB
L3 缓存：                        84 MiB
NUMA 节点0 CPU：                 0-27
NUMA 节点1 CPU：                 28-55
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; Load fences, usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
标记：                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 invpcid_single intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq md_clear pconfig spec_ctrl intel_stibp flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.24.4
[pip3] nvidia-nccl-cu11==2.20.5
[pip3] onnx==1.15.0
[pip3] paddle2onnx==1.1.0
[pip3] torch==2.3.0+cu118
[pip3] torchaudio==2.3.0+cu118
[pip3] torchtext==0.5.0
[pip3] torchvision==0.18.0+cu118
[pip3] triton==2.3.0
[pip3] tritonclient==2.19.0
[conda] Could not collectROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    NIC0    NIC1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV8     SYS     SYS     28-55   1               N/A
GPU1    NV8      X      SYS     SYS     28-55   1               N/A
NIC0    SYS     SYS      X      PIX
NIC1    SYS     SYS     PIX      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1

🐛 Describe the bug

The startup command is

CUDA_VISIBLE_DEVICES=0,1 python3 -m vllm.entrypoints.openai.api_server   --model /workspace/DeepSeek-V2-Lite-Chat/         --served-model-name atom  --gpu-memory-utilization 0.6 -tp 2 --trust-remote-code --max-model-len 10240 --enforce-eager

the error info is:

(VllmWorkerProcess pid=3773) INFO 07-10 03:56:15 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 07-10 03:56:16 custom_all_reduce_utils.py:202] generating GPU P2P access cache in /root/.config/vllm/gpu_p2p_access_cache_for_0,1.json
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method init_device: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 1800000ms for recv operation to complete, Traceback (most recent call last):
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/vllm/worker/worker.py", line 126, in init_device
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     init_worker_distributed_environment(self.parallel_config, self.rank,
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/vllm/worker/worker.py", line 327, in init_worker_distributed_environment
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/vllm/distributed/parallel_state.py", line 915, in ensure_model_parallel_initialized
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     initialize_model_parallel(tensor_model_parallel_size,
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/vllm/distributed/parallel_state.py", line 883, in initialize_model_parallel
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     _TP = init_model_parallel_group(group_ranks,
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/vllm/distributed/parallel_state.py", line 729, in init_model_parallel_group
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     return GroupCoordinator(
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/vllm/distributed/parallel_state.py", line 185, in __init__
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     self.ca_comm = CustomAllreduce(
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 126, in __init__
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     if not _can_p2p(rank, world_size):
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 30, in _can_p2p
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     if not gpu_p2p_access_check(rank, i):
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 231, in gpu_p2p_access_check
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     get_world_group().barrier()
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/vllm/distributed/parallel_state.py", line 657, in barrier
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     torch.distributed.barrier(group=self.cpu_group)
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     return func(*args, **kwargs)
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]   File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 3690, in barrier
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226]     work.wait()
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226] RuntimeError: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 1800000ms for recv operation to complete
(VllmWorkerProcess pid=3773) ERROR 07-10 04:26:16 multiproc_worker_utils.py:226] 
^C(VllmWorkerProcess pid=3773) INFO 07-10 05:49:10 multiproc_worker_utils.py:237] Worker exiting

On the same card(0,1), vllm 0.5.1+cu124 and vllm 0.5.0.post1+cu118 work fine @youkaichao Hope to get your help