tensorpipe icon indicating copy to clipboard operation
tensorpipe copied to clipboard

Make SHM support arbitrarily large libnop objects

Open lw opened this issue 5 years ago • 1 comments

In #58 and #97 we created a fastpath in SHM to serialize and deserialize protobufs directly to and from the ringbuffer. However, due to a limitation in protobuf's zero copy streams, we cannot do so when the serialized protobuf would be larger than the whole ringbuffer. So, at the moment, we fail in those cases. This will typically not be a problem, as under normal operation the protobufs we send are tiny (a few hundred bytes) and the buffer is huge (2MiB). However, a careless user could put large data in a message's metadata, which would be embedded in the protobuf and cause a failure. It would be nice if we could now re-implement a slowpath in SHM that first serializes those large protobufs (and only those!) to a temporary buffer and then chunks that buffer over many writes.

lw avatar Mar 09 '20 09:03 lw

This limitation still holds after our migration from protobuf to libnop.

lw avatar Mar 15 '21 09:03 lw