ucx
ucx copied to clipboard
UCP/WIREUP: Add capability to separate RMA and AM lanes
What
Add UCT flag when TX (like AM) depends on RX (like RDMA_READ). Implement AM lane isolation wrt to RMA lane.
Why ?
To avoid RX throughput dependency that limits overall TX throughput.
How ?
- Add a TX/RX dependency flag
- Allow skipping AM lane when creating RMA lanes
- To maintain overall number of lanes, allow overriding dev_num_paths, when a AM lane has been skipped for that reason
Tested
- Old and new version: each side performs according to its code version
- CX7 sees 1 AM + 2 RMA lanes, CX6 sees 1 lanes AM/RMA
ep 0x7f4224a4b000: am_lane 0 wireup_msg_lane <none> cm_lane <none> keepalive_lane <none> reachable_mds 0x7
ep 0x7f4224a4b000: lane[0]: 0:dc_mlx5/mlx5_0:1.0 md[0] -> addr[0].md[0]/ib/sysdev[1] seg 4294967295 am am_bw#0
ep 0x7f4224a4b000: lane[1]: 4:dc_mlx5/mlx5_2:1.0 md[2] -> addr[4].md[2]/ib/sysdev[3] seg 4294967295 rma_bw#0
ep 0x7f4224a4b000: lane[2]: 2:dc_mlx5/mlx5_1:1.0 md[1] -> addr[2].md[1]/ib/sysdev[2] seg 4294967295 rma_bw#1
ep 0x7f4224a4b000: lane[3]: 0:dc_mlx5/mlx5_0:1.1 md[0] -> addr[0].md[0]/ib/sysdev[1] seg 4294967295 rma_bw#4
ep 0x7f4224a4b000: lane[4]: 4:dc_mlx5/mlx5_2:1.1 md[2] -> addr[4].md[2]/ib/sysdev[3] seg 4294967295 rma_bw#2
ep 0x7f4224a4b000: lane[5]: 2:dc_mlx5/mlx5_1:1.1 md[1] -> addr[2].md[1]/ib/sysdev[2] seg 4294967295 rma_bw#3
ep 0x7f4224a4b000: lane[6]: 0:dc_mlx5/mlx5_0:1.2 md[0] -> addr[0].md[0]/ib/sysdev[1] seg 4294967295 rma_bw#5
^---- added lane[6]