hpx icon indicating copy to clipboard operation
hpx copied to clipboard

Implement zero-copy serialization for action arguments

Open hkaiser opened this issue 11 years ago • 3 comments

If an argument for an action is bitwise copy able (like std::vector<T> for T being bitwise copy able) it should be possible to avoid to copy it into (and out of) the serialization buffer. Investigate whether this is actually implementable.

The following tasks need to be resolved:

  • [x] Implement support in the serialization archives to store data not only in a buffer, but to maintain a list of chunks (non-bitwise serializable data ends up in the main serialization buffer as before) (see: ecafc9d669bc143a9de4c7a5177b364814af6c63)
    • It turned out not to be possible to have zero-copy de-serialization as the buffers for the data to de-serialize are allocated inside the serialization code only after the data has been received and before the archives are invoked. Zero-copy serialization was implemented, though.
  • [x] Implement test verifying this serialization functionality (see: ecafc9d669bc143a9de4c7a5177b364814af6c63)
  • [ ] Pass the buffer and the list of chunks to the used networking layer (use scatter/gather read/writes where possible):
    • [x] TCP/IP: Boost.Asio supports this by using asio::buffer (done in 268565dd1fe265573e7b4da03a972ad89d368585)
    • [ ] RDMA/IB: scatter/gather read/writes are directly supported
    • [x] MPI: this could turn out to be tricky as MPI_Isend/MPI_Irecv have no direct support for this (done with ea72168f193ae16bf7b434a831b24e43107c3522)

hkaiser avatar Aug 02 '13 02:08 hkaiser

The first two tasks have been merged to the master branch here: 870e507e28f77caaf5073e949fa7fe010456e592

hkaiser avatar Sep 10 '13 01:09 hkaiser

Moving to 1.0.0 since it is not critical for the next release.

sithhell avatar Oct 17 '13 13:10 sithhell

50bededc1ab7b3fe12fda3d1d1b98ada823d5b08 implements zero-copy deserialization for the type util::serialize_buffer.

hkaiser avatar Mar 30 '14 01:03 hkaiser