Serializing/Deserialzing Large Arrays - Length header size is larger than buffer size

[dexsper]

I use MemoryPack to save building information in the game world. And everything works fine usually, hundreds of people downloading and saving their builds at the same time. But when I wanted to migrate and add a new field manually (deserialize old format -> serialize to new format) I started getting a header size error:

Unhandled exception. MemoryPack.MemoryPackSerializationException: Length header size is larger than buffer size, length: 218103807.

Serializing:

List<EntityData> entitiesData = ...
using var compressor = new BrotliCompressor();
MemoryPackSerializer.Serialize(compressor, entitiesData); 

Deserializing:

byte[]? data = ...
using var decompressor = new BrotliDecompressor();
MemoryPackSerializer.Deserialize(decompressor.Decompress(data), ref entitiesData!);

I don't have the ability to use streaming serialization like. And during migration I need to deserialize about 10000 files (~150 kilobytes each) and serialize again

[dexsper]

It happens at the point when it comes to deserializing textures in some files

[neuecc]

Thank you. It looks like there's a problem with the compression. I'll check it out.

[dexsper]

I don't know if this will help you to fix the error, but I managed to avoid it by checking that the array is empty, having previously marked Payload and Textures as nullable - apparently at some point an array with size 0 was created.

[79loli]

Length header size is larger than buffer size, length: 8704.

[79loli]

Specified argument was out of the range of valid values. Parameter name: capacity