CompositeByteBuf memory allocation for Frame messages

[agusevas]

Motivation

I have been evaluating the RSocket and did some profiling. It looks like creating Frame message many ByteBuf buffers are used (i.e. header, payload, frame size). Those buffers are composed using CompositeByteBuf. The ByteBuf can be reused, but the CompositeByteBuf are always created as new ones. Such CompositeByteBuf allocation doesn't take much memory, but sending many Frame messages it sums up. Here is an example memory allocation using async-profiler:

Also when Netty sends the Frame messages it needs to duplicate internal DirectByteBuffer objects because of CompositeByteBuf usage:

This wouldn't be the case if only one ByteBuf were used instead. As far as I understand, it is done in order to avoid copying buffers. Nevertheless maybe there is a way of improving it?

Considered alternatives

Alternatives would be:

• avoid using CompositeByteBuf and just use one ByteBuf per Frame message.
• provide a way of customize the creation of Frame messages. Current implementation is using static SendUtils.sendReleasingPayload() method and there is no way of replacing it.

Additional context

I'm using rsocket-java version 1.1.2 with TCP transport.

[OlegDokuka]

@agusevas thanks for your investigation!

I think the best way to go is to create a similar issue at the netty project and reference this one! We can contribute to netty if needed