bifrost icon indicating copy to clipboard operation
bifrost copied to clipboard
verified publisher icon ledatelescope

`ibverb-support` fails under OFED 5.5

Open jaycedowell opened this issue 2 years ago • 2 comments

After upgrading a machine to Ubuntu 20/OFED 5.5 I'm running into problems with the ibverb-support branch, mainly segfaults.

jaycedowell avatar Jan 18 '23 16:01 jaycedowell

This could just be user error on my part.

jaycedowell avatar Jan 18 '23 16:01 jaycedowell

I think everything is working now. My problems were:

  • outdated version of Bifrost Verbs support -> what's in ibverb-support does work
  • problems with the OFED installation -> there were some old /etc/modprobe.d config. files that caused mlx5_core to be loaded with the wrong options
  • changes to the permissions model of OFED -> the Verbs support that we use now requires the CAP_NET_RAW capability to work.

For this last point I ended up going to the spead2 documentation for help:

  • https://spead2.readthedocs.io/en/latest/py-ibverbs.html
  • https://spead2.readthedocs.io/en/latest/tools.html#spead2-net-raw

We should try to convey this information to users of Bifrost as well.

jaycedowell avatar Feb 10 '23 22:02 jaycedowell

Should be fixed with #248.

jaycedowell avatar Sep 06 '24 01:09 jaycedowell