ib_write_bw -a or not？

[tensorflowt]

I am currently testing the write RDMA bandwidth of the IB network card. I used two methods to test it: The first：

#server
watch ib_write_bw  -d mlx5_0 -q 1 --report_gbits -F

#client
ids=mlx5_0
ip=192.168.100.201
for id in ${ids//,/ }
do  
   ib_write_bw  -d $id -q 1 --report_gbits  -F $ip
   sleep 3
done

The test results are as follows：

The second：

#server
watch ib_write_bw  -d mlx5_0 -q 1 --report_gbits -F -a

#client
ids=mlx5_0
ip=192.168.100.201
for id in ${ids//,/ }
do  
   ib_write_bw  -d $id -q 1 --report_gbits  -F -a $ip
   sleep 3
done

The test results are as follows：

From the test results of method 1, the indicators are not in line with expectations, because its rate is 400 Gb/sec (4X NDR) But it seems normal from the second method. Which one should I follow?

[sshaulnv]

it is weird, the results should be similar... i cant reproduce this behavior over my setups, can you please share the results of:

1. run method 1 with more iterations.
2. run method 1 with larger msg size.