cbornet
Study performance improvements
Compared to Pulsar's binary protocol, gRPC + protobuf has some drawbacks regarding performance and possibility to do zero-copy and object recycling
On the receiving side, the payload is mapped to a ByteString. It seems we could access the underlying buffer with protobuf's UnsafeByteOperations but gRPC will release its Netty buffer right after the onNext execution so we can't retain it until the write to the ledger has finished. For this we would probably need to get the reference to gRPC's ByteBuf so we can retain it and release it later (there are actually multiple buffers aggregated into a CompositeReadableBuffer)
On the sending side, Protobuf by nature doesn't map the message representation to the serialized bytes (unlike Flatbuffer or Cap'n Proto). So there will always be a copy from the message to the serialized output. We currently have another copy from the ledger entry ByteBuf to the Protobuf ByteString because we don't know when gRPC has finished using the buffer and we need to release the ByteBuf. It would be nice if we could register a callback that is called when gRPC no longer uses the buffer. See discussion in https://github.com/grpc/grpc-java/issues/1054#issuecomment-631126609
Note : this protocol handler being developed to be an easier-to-use version of the binary protocol and not a replacement, it should make life as easy as possible for the clients and use only standard gRPC + Protobuf. So even if it could be possible to replace Protobuf by another content-type (flatbuffer, cap'n proto or a custom framing like the one used in the binary protocol), it should probably not be done at the moment and we should accept the performance penalty.
What if we use Netty HTTP/2 handler and then parse protobuf binary data using direct memory instead of using gRPC-java?
Yes that would allow some improvement with the lifecycle management. It would be a lot of work to implement correctly the gRPC protocol though. Certainly doable but I don't think I have the strength. The protobuf serialization issue would remain.