ucx icon indicating copy to clipboard operation
ucx copied to clipboard

UCP/WIREUP: Add capability to separate RMA and AM lanes

Open tvegas1 opened this issue 1 year ago • 0 comments

What

Add UCT flag when TX (like AM) depends on RX (like RDMA_READ). Implement AM lane isolation wrt to RMA lane.

Why ?

To avoid RX throughput dependency that limits overall TX throughput.

How ?

  • Add a TX/RX dependency flag
  • Allow skipping AM lane when creating RMA lanes
    • To maintain overall number of lanes, allow overriding dev_num_paths, when a AM lane has been skipped for that reason

Tested

  • Old and new version: each side performs according to its code version
  • CX7 sees 1 AM + 2 RMA lanes, CX6 sees 1 lanes AM/RMA
ep 0x7f4224a4b000: am_lane 0 wireup_msg_lane <none> cm_lane <none> keepalive_lane <none> reachable_mds 0x7
ep 0x7f4224a4b000: lane[0]:  0:dc_mlx5/mlx5_0:1.0 md[0]      -> addr[0].md[0]/ib/sysdev[1] seg 4294967295 am am_bw#0
ep 0x7f4224a4b000: lane[1]:  4:dc_mlx5/mlx5_2:1.0 md[2]      -> addr[4].md[2]/ib/sysdev[3] seg 4294967295 rma_bw#0
ep 0x7f4224a4b000: lane[2]:  2:dc_mlx5/mlx5_1:1.0 md[1]      -> addr[2].md[1]/ib/sysdev[2] seg 4294967295 rma_bw#1
ep 0x7f4224a4b000: lane[3]:  0:dc_mlx5/mlx5_0:1.1 md[0]      -> addr[0].md[0]/ib/sysdev[1] seg 4294967295 rma_bw#4
ep 0x7f4224a4b000: lane[4]:  4:dc_mlx5/mlx5_2:1.1 md[2]      -> addr[4].md[2]/ib/sysdev[3] seg 4294967295 rma_bw#2
ep 0x7f4224a4b000: lane[5]:  2:dc_mlx5/mlx5_1:1.1 md[1]      -> addr[2].md[1]/ib/sysdev[2] seg 4294967295 rma_bw#3
ep 0x7f4224a4b000: lane[6]:  0:dc_mlx5/mlx5_0:1.2 md[0]      -> addr[0].md[0]/ib/sysdev[1] seg 4294967295 rma_bw#5
   ^---- added lane[6]

tvegas1 avatar Feb 07 '24 15:02 tvegas1