flatbuffers
flatbuffers copied to clipboard
Idea: root versioning with var-uints
Users have to know what's at the root of their flatbuffer. Sometimes there's a size prefix and I've suggested the root can be a 64bit vector in https://github.com/google/flatbuffers/issues/7537#issuecomment-1252714710. There's also variability in the presence of a file prefix. I think it'd be problematic if the number of "root types" proliferated and users had to keep track of that, so I propose the following scheme so buffers can know about their own roots.
(This supposes we accept my suggestion in #7537 about having a 64-bit-vector-root type)
The first bytes are a variable length unsigned integer. We'll call it v
.
-
v & 6
will tell us the size prefix, (this comes after the varuint and may be unaligned)-
v & 6 == 0
means there is no size prefix -
v & 6 == 2
means there is a 4 byte size prefix -
v & 6 == 4
means there's an 8 byte size prefix -
v & 6 == 6
is interpreted the same asv & 6 == 4
-
-
v & 8 == 1
if there is a 4 byte file identifier after the root offset -
v & 16 == 1
means the root offset is to a 64-bit vector and not the normal offset to the root table
This is probably overkill and we'd have to be careful about whether we promise forwards/backwards compatibility and what unknown versions mean, but I'd like to hear what people think.
Could we give thought on a having it implemented as a normal flatbuffer table
instead?
For Example:
table metatable {
len:int64;
file_identifier:string;
root_offset:int64;
etc...
}
That way we can default them in code to sensible defaults, but allow users to customize them?
The pros
are:
- We can use existing code for accessing these fields
- Allows future expansion and forwards/backwards compatibility
The cons
are:
- Bloated size, since we now need at least
SOffset32
+ 2uint16_t
more space for thismetatable
- Not sure how we can implement this seamlessly to work with older data.
We'd probably want lazy or shallow verification if we recommend that, since this metadata is most useful when you're not sure you actually have a flatbuffer table. Also, yeah there's nothing that can be done about forwards/backwards compatibility, I think we're cornered on that one
This issue is stale because it has been open 6 months with no activity. Please comment or label not-stale
, or this will be closed in 14 days.
This issue was automatically closed due to no activity for 6 months plus the 14 day notice period.