perftest
perftest copied to clipboard
linux-rdma
perftest_parameters: Enable RDMA atomic with ODP feature
RDMA atomic with ODP is already enabled in the mlx5 driver. There is no reason to block the feature. cf. https://github.com/torvalds/linux/commit/17d2f88f92ce39b348f125f6b2e6eeb6b0906ac7
Signed-off-by: Daisuke Matsuda [email protected]
Hi @daimatsuda , Thanks for your contribution! you are right regarding the mlx5. but I see that removing the if condition that blocks the odp with atomic will result in a wrong fail message for mlx4 devices (for example cx3pro) "Send is not supported for RC transport. failed to create mr Failed to create MR Couldn't create IB resources"
Need to find a way to enable/disable this feature according to the device support of odp
Hello @HassanKhadour, thank you for taking a look.
but I see that removing the if condition that blocks the odp with atomic will result in a wrong fail message for mlx4 devices (for example cx3pro)
Nice catch! I did some tests using the rxe driver, which does not have the ODP feature either, and found that the wrong message is generated not only with atomic but also with other operations when odp flag is specified.
Need to find a way to enable/disable this feature according to the device support of odp
I have added another patch to handle this issue.
Well, I found the way to fetch xrc odp caps and that there are some points to be improved. I will repost the third patch today.
I think now the patches are OK. However, there seems a problem with the mlx5 driver, firmware, or hardware. I am using "Mellanox Technologies MT28908 Family [ConnectX-6]". The driver fills the capabilities at internal_fill_odp_caps(). cf. https://github.com/torvalds/linux/blame/master/drivers/infiniband/hw/mlx5/odp.c#L321
The problem is that some capabilities are not filled while the operation can be used with ODP. For example, I can use RC ATOMIC with ODP and can observe ODP code is working, but IB_ODP_SUPPORT_ATOMIC(0x10) is not set in the capability(rc_odp_caps). Here are the capabilities printed at the bottom of internal_fill_odp_caps():
rc_odp_caps=2f / uc_odp_caps=0 / ud_odp_caps=1 / xrc_odp_caps=2d
It seems RC ATOMIC, UD SRQ_RECV, XRC RECV, XRC ATOMIC are left unfilled. I doubt if MLX5_CAP_ODP() macro is working correctly with all capabilities, or the firmware may not configure the values properly. I have no time nor ideas to investigate this further right now. Do you have any idea about this problem?
Anyway, I believe checking ODP capabilities is necessary procedure. Merging the patches would not bother anybody because ODP ATOMICs are not available in perftest from the beginning. If the problem is reproducible on other environments, then it should be addressed by the vendor.
I have added another block to check the bidirectional case. Thanks for your suggestion!