RDMA Read Error on DGX B200 systems

[johnmoore4]

We are running an application on a single node using CUDA 12.8. We are using openMPI 5.0.7 with UCS 1.8.0 and gdrcopy 1.5.0 and nv_peer_mem v 1.3.

We are getting the error below: ib_mlx5_log.c:179 Remote operation error on mlx5_4:1/IB (synd 0x14 vend 0x89 hw_synd 0/0) ib_mlx5_log.c:179 RC QP 0x53 wqe[0]: RDMA_READ s-- [rva 0x7fe7d1a00000 rkey 0x180ac9] [va 0x7f15bfe00000 len 10020 lkey 0x182fee] [rqpn 0x5f dlid=65 sl=0 port=1 src_path_bits=0] ==== backtrace (tid: 6594) ==== 0 /shared_data/third_party/openmpi-5.0.7/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x7f18ebadf934] 1 /shared_data/third_party/openmpi-5.0.7/lib/libucs.so.0(ucs_fatal_error_message+0xc2) [0x7f18ebadc9c2] 2 /shared_data/third_party/openmpi-5.0.7/lib/libucs.so.0(ucs_log_default_handler+0xf7e) [0x7f18ebae15ee] 3 /shared_data/third_party/openmpi-5.0.7/lib/libucs.so.0(ucs_log_dispatch+0xe4) [0x7f18ebae1a14] 4 /shared_data/third_party/openmpi-5.0.7/lib/ucx/libuct_ib_mlx5.so.0(uct_ib_mlx5_completion_with_err+0x60d) [0x7f160210036d] 5 /shared_data/third_party/openmpi-5.0.7/lib/ucx/libuct_ib_mlx5.so.0(uct_rc_mlx5_iface_handle_failure+0x134) [0x7f1602115f24] 6 /shared_data/third_party/openmpi-5.0.7/lib/ucx/libuct_ib_mlx5.so.0(uct_ib_mlx5_check_completion+0x3d) [0x7f160210153d] 7 /shared_data/third_party/openmpi-5.0.7/lib/ucx/libuct_ib_mlx5.so.0(+0x2b2f7) [0x7f16021172f7] 8 /shared_data/third_party/openmpi-5.0.7/lib/libucp.so.0(ucp_worker_progress+0x2a) [0x7f18ebb6c05a] 9 /shared_data/third_party/openmpi-5.0.7/lib/libopen-pal.so.80(opal_progress+0x34) [0x7f18ebc2e734] 10 /shared_data/third_party/openmpi-5.0.7/lib/libmpi.so.40(ompi_request_default_wait+0x140) [0x7f1904112020] 11 /shared_data/third_party/openmpi-5.0.7/lib/libmpi.so.40(MPI_Wait+0x54) [0x7f190415de84]

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

5.0.7, UCX 1.8.0

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

OpenMPI was compiled from source

Please describe the system on which you are running

• Operating system/version: Ubuntu 22.04.4 LTS
• Computer hardware: NVIDIA DGX B200
• Network type: Infiniband NDR

---

Details of the problem

The command we are issuing is:

/third_party/openmpi-5.0.7/bin/mpirun -np 8 -hostfile ./hostfile --report-bindings --bind-to core --map-by ppr:8:node:PE=14 --mca pml ucx --mca btl ^openib -x UCX_TLS=self,sm,cma,cuda_copy,gdr_copy,rc_v -x UCX_IB_GPU_DIRECT_RDMA=1 ./mpi_rail_mapping_b200.sh /install_path/openmpi507-25.6.0/bin/OurExecutable

and the contents of mpi_rail_mapping.sh is:

#!/bin/bash export LOCAL_RANK=$OMPI_COMM_WORLD_LOCAL_RANK IB_DEVS=(4 7 8 9 10 13 14 15) CUDA_DEV=$LOCAL_RANK IB_DEV=${IB_DEVS[$LOCAL_RANK]} export UCX_NET_DEVICES=mlx5_$IB_DEV:1 echo "local rank $CUDA_DEV: using hca $IB_DEV" exec $*

The IB_DEVS is set to map each GPU to the nearest IB card according to the output of nvidia-smi topo -m shown below:

nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 NIC11 NIC12 NIC13 NIC14 NIC15 CPU Affinit y NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE PXB NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS 0-55 0 N /A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE NODE NODE NODE PXB NODE NODE SYS SYS SYS SYS SYS SYS 0-55 0 N /A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE NODE NODE NODE NODE PXB NODE SYS SYS SYS SYS SYS SYS 0-55 0 N /A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE NODE NODE NODE NODE NODE PXB SYS SYS SYS SYS SYS SYS 0-55 0 N /A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PXB NODE NODE NODE NODE NODE 56-111 1 N /A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE PXB NODE NODE 56-111 1 N /A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE PXB NODE 56-111 1 N /A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE PXB 56-111 1 N /A NIC0 NODE NODE NODE NODE SYS SYS SYS SYS X PIX PIX PIX NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS NIC1 NODE NODE NODE NODE SYS SYS SYS SYS PIX X PIX PIX NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS NIC2 NODE NODE NODE NODE SYS SYS SYS SYS PIX PIX X PIX NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS NIC3 NODE NODE NODE NODE SYS SYS SYS SYS PIX PIX PIX X NODE NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS NIC4 PXB NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE NODE X NODE NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS NIC5 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE NODE NODE X PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS NIC6 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE NODE NODE NODE PIX X NODE NODE NODE SYS SYS SYS SYS SYS SYS NIC7 NODE PXB NODE NODE SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE X NODE NODE SYS SYS SYS SYS SYS SYS NIC8 NODE NODE PXB NODE SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODE X NODE SYS SYS SYS SYS SYS SYS NIC9 NODE NODE NODE PXB SYS SYS SYS SYS NODE NODE NODE NODE NODE NODE NODE NODE NODE X SYS SYS SYS SYS SYS SYS NIC10 SYS SYS SYS SYS PXB NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X NODE NODE NODE NODE NODE NIC11 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE X PIX NODE NODE NODE NIC12 SYS SYS SYS SYS NODE NODE NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE PIX X NODE NODE NODE NIC13 SYS SYS SYS SYS NODE PXB NODE NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE X NODE NODE NIC14 SYS SYS SYS SYS NODE NODE PXB NODE SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE X NODE NIC15 SYS SYS SYS SYS NODE NODE NODE PXB SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE X

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NIC8: mlx5_8 NIC9: mlx5_9 NIC10: mlx5_10 NIC11: mlx5_11 NIC12: mlx5_12 NIC13: mlx5_13 NIC14: mlx5_14 NIC15: mlx5_15

If we permute the IB_DEV mapping, to say (4,8,9,10,13,14,15,7) it will run, but I assume that is because the messages are going through the NUMA node, and no longer using GPU RDMA.

Also, if I add cuda_ipc to the UCX_TLS it will work, but this is because it is using NVLINK instead of the IB cards.

Any advice would be appreciated.

[janjust]

@Akshay-Venkatesh ^ any advice?