Wrap FB object inside FB object without copy?

[ryan-motive]

I'm trying to write a protocol with FlatBuffers that has separate transport layer from the messaging layer. I'd like to use FlatBuffers for both layers. The goal is for a transport relay server to be able to read and manipulate the outer layer with as little effort as possible, and without having to manipulate or read the inner message. Ideally the whole thing could be done on both client & server with as few memory copies as possible. I'm using C#.

First I tried a schema that looks roughly like this:

table TransportMessage {
    sender_id: int;
    another_header: int;
    message: InnerMessage;
}

table InnerMessage { ... }

The goal is for the server to update another_header before forwarding it. However with this format, I can't find any easy way to either update another_header directly in the buffer (I can understand why this probably isn't supported) or to copy "message" verbatim over to another buffer. If there was some way to easily get the InnerMessage and drop it into a new FlatBufferBuilder I could live with that.

The next approach I tried without luck was more like this:

table TransportMessage {
    sender_id: int;
    another_header: int;
    message: [byte];
}

My hope here was that I could essentially write the whole "Message" first into FlatBufferBuilder, then just point the message vector at its offset. But this didn't work using the APIs available (may have missed something here though). The idea with this one is that I'd at least get direct access to the message bytes on the relay server and could copy them that way.

The only way I can see to do this with the public API is to write the full message into a byte array, then copy that array into a new FlatBufferBuilder. This just seems like a lot of overhead though since the first pass already did all the hard work of copying the data into a buffer and a main goal of FlatBuffers is to avoid these sorts of copies. I just need to coerce the builder to point at this buffer offset. I tried creating a second FlatBufferBuilder with the same buffer, but of course it resets the ByteBuffer as its first step so loses that data. I was hoping there was maybe a "FastForward" type of call that would manually update the offset, but didn't see one.

I can see that it wouldn't be too hard to make our own writer that would be able to do this, but I'm just wondering if I'm missing an approach with the existing API that could achieve this.

[mzaks]

See nested_flatbuffer Attribute https://google.github.io/flatbuffers/flatbuffers_guide_writing_schema.html

[ryan-motive]

Thanks @mzaks - this definitely simplifies the accessor side, but it looks like it suffers from the same issue on the writing side. Seems like the only option is a custom FBB that lets me point to an existing buffer? I think the only step I need to add is to prepend the vector length?

[CasperN]

On the writing side, your message and transport layers can share the same flatbuffer builder - when the message layer finishes the message it can pass its wip table offset to the transport layer to finish.

This ownership story is a little complex but would that work for you?

[ryan-motive]

So yeah - it's already doing essentially this (I added a "wrap" step that writes the headers). Ownership is a little complex, but I think straightforward enough to not be unusable. The only issue I've found is that I can't seem to link the Transport message field to the buffer for the Message itself--i.e. I don't see an easy way to produce a "VectorOffset" that's configured properly. But I think just injecting the vector size will do it?

[ryan-motive]

Yup - "PutInt"-ing the size and then using that offset in a subsequent FlatBuffer seems to have done the trick. 👍

[github-actions[bot]]

This issue is stale because it has been open 6 months with no activity. Please comment or label not-stale, or this will be closed in 14 days.

[github-actions[bot]]

This issue was automatically closed due to no activity for 6 months plus the 14 day notice period.