Merge SC19 LibFabric Parcelport fixes into Master

[aserio]

The branch: https://github.com/STEllAR-GROUP/hpx/commits/rdma_object needs to be merged into master however this requires considerable work to get it into shape.

• [ ] rdma_object<T> , rdma_vector<T>

  1. Needs additional touching up and potentially should be integrated into the channel<T> abstraction
  2. rdma_object and rdma_vector are new types of that do not work with the current tcp parcelport. (so they become 'conditional' - i.e. they can only be used when the LF PP is enabled which means that code written to target the new features would not compile/run with the default TCP parcelport).
  3. The rdma types expose pinned memory to the user that is allocated from the parcelport layer and then consumed by the serialization layer to elide on the fly memory registrations. The TCP parcelport could be extended to expose memory in a similar way, however making use of the libfabric TCP provider would be a better way of supporting this (see below).

• [ ] Merge current changes with recent changes to the parcelport

  1. This branch required major changes in a lot of low level serialization and memory management (registration/pinning for RDMA operations) code which needs to be merged with master.

• [ ] Remove existing TCP parcelport

  1. The existing TCP parcelport requires certain (internal) API abstractions that make the libfabric parcelport implementation more complex than it needs to be. The libfabric API meshes quite cleanly with HPX internals. Removal of the TCP parcelport would simplify the entire parcelport codebase significantly.
  2. The libfabric parcelport has been tested using the libfabric TCP provider and can now be used on linux machines as a replacement for the existing TCP parcelport, so one major obstacle has been removed. Unfortunately, the need to support the windows operating system would make it difficult at the current time to do away with the TCP parcelport completely. There is support for libfabric on windows, but this has not been investigated and would probably require considerable work.
  3. Getting libfabric on windows running smoothly (irrespective of HPX integration) might make a good GSoC project. Adding CMake support, building and testing on windows and adding libfabric as a (sub)module in HPX with build integration would make a worthwhile project.
  4. The rdma_object<> and rdma_vector<> abstractions have been tested with the libfabric TCP provider and are supported - the TCP backend emulates RDMA put/get operations and exposes a uniform API so that the new constructions created for HPX can be used even on legacy networks.

The above list is not exhaustive and other changes may also be necessary.

[biddisco]

List of tasks has been updated/extended.

[biddisco]

I asked about windows support on the libfabric mailing list and received the reply

Windows is supported. You should be able to run the networkdirect, tcp, sockets, and udp providers (plus the utility providers over those). NetworkDirect is for IB/iWarp/RoCE devices (think of it as libibverbs for Windows).

We run smoke tests on windows as part of all pull requests, but not more involved validation. However, since the same code is used on Linux and windows, there's a good chance it works well. We also don't create windows packages for libfabric. The windows build projects are stored in the git repo.

This means that in principle, we could entirely replace the existing TCP boost:asio based parcelport with a sockets implementation on windows - as noted previously, this would reduce code size by allowing a single parcelport implementation across a wide range of devices and reduce boost dependencies as well as making new rdma based types functional without requiring #ifdefs or conditional compilation of certain classes.

[biddisco]

Just want to mention that the latest libfabric has early support for collectives https://ofiwg.github.io/libfabric/master/man/fi_collective.3.html this is one (more) reason why moving from a tcp (boost/std::asio) based default to a libfabric based one would be a good move.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.