flatbuffers icon indicating copy to clipboard operation
flatbuffers copied to clipboard

[C++] Fix vtable deduplication for 64-bit buffers >2GB

Open gowsiany opened this issue 7 months ago • 0 comments

Fixes #8590

Summary

This PR corrects vtable deduplication for FlatBuffers using 64-bit offsets. Previously, when serializing buffers larger than 2GB, vtable deduplication failed, resulting in much slower serialization and larger buffer sizes.

Root Cause

The issue was caused by 64-bit offset calculations not accounting for length_of_64_bit_region_. As a result, the addresses of vtable candidates stored in scratch_data were miscalculated, preventing proper deduplication.

Solution

This fix includes length_of_64_bit_region_ in the offset calculation for vtables in 64-bit mode. This ensures correct vtable candidate addresses, enabling deduplication and restoring efficient serialization for large buffers.

Impact

  • Restores vtable deduplication for buffers >2GB with 64-bit offsets.
  • Significantly improves serialization performance and reduces buffer size for large data sets.
  • Maintains full backward compatibility with 32-bit buffers.

Verification

  • Serialization benchmarks with large (2GB+) buffers now show linear scaling and reduced buffer size.

gowsiany avatar May 02 '25 10:05 gowsiany