[Rust] Proposal for non-owning FlatBufferBuilder API

[bsilver8192]

I'd like to use a FlatBufferBuilder directly into an externally-allocated buffer from Rust.

Some context: I'm looking to implement the equivalent of this C++ API. It's an interface to a shared memory IPC system which builds a table in a region of shared memory, before handing off ownership to other processes (after the table is completed).

The C++ version has a flatbuffers::Allocator implementation which aborts if you do anything besides a single allocate call. Using the initial_size argument to FlatBufferBuilder's constructor results in the FlatBufferBuilder allocating that single piece of memory and then filling it in, and if you try building a table that's too large it aborts in the custom Allocator implementation with an error message pointing to the size that needs to be increased.

This is part of what's needed for #7005.

Proposed API:

/// TODO: Does the memory need to be zeroed?
trait Allocator: Index<SliceIndex<[u8]>> + IndexMut<SliceIndex<[u8]>> {
    /// Grows the buffer, with the old contents being moved to the end.
    ///
    /// Returns the amount it has grown by, so the caller can update offsets.
    fn grow_downwards(&mut self) -> usize;
    fn len(&self) -> usize;
}

struct DefaultAllocator {
    owned_buf: Vec<u8>,
}
/// len, Index, and IndexMut forward to owned_buf.
/// grow_downwards is the current FlatBufferBuilder::grown_owned_buf, except the self.used_space()
/// and self.head operations which stay in grow_owned_buf.

pub struct FlatBufferBuilder<A: Allocator, 'fbb> {
    buf: A,
    <everything else the same>
}

/// Not sure if this works directly, or if it needs a marker trait that only DefaultAllocator implements.
impl<A: DefaultAllocator, 'fbb> FlatBufferBuilder<A, 'fbb> {
    pub fn with_capacity(size: usize) -> Self {
        Self::with_capacity_and_allocator(size, DefaultAllocator::new())
    }
    /// `new` moves here too.
}

A more direction translation of the C++ API would be something like this:

enum SomeAllocator {
    Default(Vec<u8>),
    Dyn(Box<dyn Allocator>),
}

but I don't think that makes sense in Rust. Rust's generics are easier to work with than C++, so doing it with the trait directly means users can have a custom allocator without any vtables if desired. It also allows the custom type to carry lifetimes with it, which I want for my use case.

I'm prepared to implement this and send a PR. Thoughts on the API before I start doing that?

[CasperN]

At a high level, having an alllocator trait with a default allocator based on Vec looks good to me. I agree it's more Rust-y than the enum approach.

Note that we want to maintain backwards compatibility with existing users' code so FlatBufferBuilder should not be changed to require any parameters. This can be achieved with a simple wrapper like struct FlatBufferBuilder(FlatBufferBuilderInner<DefaultAllocator>);, or maybe there's something more elegant that can be done.

I wonder if we should bubble up OOMs. Panic/abort is the easiest way to implement this, but maybe someone would want a try_ API. Food for thought, but no need to implement this until someone asks for it.

I'm a bit curious as to why it's needed for #7005 though, can you expand on that part?

[bsilver8192]

Note that we want to maintain backwards compatibility with existing users' code so FlatBufferBuilder should not be changed to require any parameters. This can be achieved with a simple wrapper like struct FlatBufferBuilder(FlatBufferBuilderInner<DefaultAllocator>);, or maybe there's something more elegant that can be done.

Agreed. Thoughts on using a new type for FlatBufferBuilder vs a type alias? The type alias avoids needing to either to manually write forwarding methods or have a Deref which has some confusing interactions.

I'm a bit curious as to why it's needed for #7005 though, can you expand on that part?

If you can bound the maximum size of the nested flatbuffer, you can allocate its space in the parent flatbuffer and then use this API to build the nested flatbuffer in it. I don't think the Rust API exposes a way to do that, but the C++ API does.

[CasperN]

Thoughts on using a new type for FlatBufferBuilder vs a type alias?

Type alias seems good to me

If you can bound the maximum size of the nested flatbuffer, you can allocate its space in the parent flatbuffer and then use this API to build the nested flatbuffer in it.

Thanks

[github-actions[bot]]

This issue is stale because it has been open 6 months with no activity. Please comment or label not-stale, or this will be closed in 14 days.

[bsilver8192]

I've been busy with other projects, but I am still considering doing this when I get back to it.

[github-actions[bot]]

This issue is stale because it has been open 6 months with no activity. Please comment or label not-stale, or this will be closed in 14 days.

[github-actions[bot]]

This issue was automatically closed due to no activity for 6 months plus the 14 day notice period.

[adsnaider]

I went ahead and implemented the Allocator trait in #8106