HG Bulk: Improve non-contiguous bulk transfers

[soumagne]

E-mail from Xuezhao:

1. Is it considerable to add a range list parameter to HG_Bulk_transfer API? (or by adding another HG_Bulk_transferv() API to keep current API not changed) The bulk handle possible associated with a set of buffer segments like [addr, addr+2M], [addr+4M, addr+6M], ... In some complex data transferring mode, users might need to do bulk transferring to [addr+1M, addr+2M], [addr+4M, addr+5M], ... With current API, for above usage user can-only initiate multiple bulk transferring. If it can pass in a range list parameter (a set of "offset, len pairs), and user can be more convenient to use (and potentially be more efficient if the data can be transferred by one RMA operation). At ofi level the fi_readv/writev can do that kind of fragmented RMA.

2. HG_Bulk_create Good to see that the mem_handle_create_segments() added to NA callback to allow create one NA mem handle for multiple buffer segments. One detail is that for OFI there is a limitation of how many iov segments is allowed for fi_mr_regv(), and it possible be different for different providers (mr_iov_limit, for socket provider it is 8, and psm2 is 65536). The current handing in hg_bulk_create() is not very convenient to bridge to OFI as it does not consider the mr_iov_limit restrict (agreed the mem_handle_create_segments callback need to be implemented to ofi plugin).

The HG bulk API must support:

• [ ] non-contiguous HG Bulk transfer calls (besides supporting the non-contiguous memory registrations)
• [ ] non contiguous memory registrations that exceed the underlying NA plugin limit (max IOV count). Need to add another NA callback to return that limit to the upper HG bulk API.