How to get around max size of 2GB for very big sequential objects?

[machineko]

I have pretty average neural network (10GB) and small exporter using flatbuffers how do i "fight" int32 offset limit in python, it is even possible or i have to switch to protobuf for that use case?

[CasperN]

There isn't a good way around it. Our size limits are generally set by constants (which you could modify in a branch) but that will limit the portability of your flatbuffer.

Note that protobuf also has a 2GB size limit

[machineko]

could you point me to places when i have to modify size limit?

[CasperN]

Just look for UOffset and SOffset in the repo. I'm going to warn you that this path isn't tested/supported, so if it doesn't work first try, then some code somewhere assumes the size of these types without referring to them which will be a huge pain to track down. That said, someone have done it before as alluded to in #4505

[dbaileychess]

Going to close since it is not directly actionable, and we already track 2 GB limit elsewhere.