tensorpipe
tensorpipe copied to clipboard
Make SHM support arbitrarily large libnop objects
In #58 and #97 we created a fastpath in SHM to serialize and deserialize protobufs directly to and from the ringbuffer. However, due to a limitation in protobuf's zero copy streams, we cannot do so when the serialized protobuf would be larger than the whole ringbuffer. So, at the moment, we fail in those cases. This will typically not be a problem, as under normal operation the protobufs we send are tiny (a few hundred bytes) and the buffer is huge (2MiB). However, a careless user could put large data in a message's metadata, which would be embedded in the protobuf and cause a failure. It would be nice if we could now re-implement a slowpath in SHM that first serializes those large protobufs (and only those!) to a temporary buffer and then chunks that buffer over many writes.
This limitation still holds after our migration from protobuf to libnop.