asterisc
asterisc copied to clipboard
Excessive memory usage serialising state snapshots
When a client program uses a large amount of memory (~3GiB in my test case), serialising the state snapshot to JSON uses an excessive amount of system memory. For a 3GiB state serialising the state consumed 64Gb of system memory and was killed by the OOM killer.
In cannon the memory pages are compressed before serialising to JSON however that's a pretty rubbish solution and is probably insufficient given asterisc is 64bit so the client memory usage can be very high.
The ideal solution would be a state snapshot format that can be streamed rather than needing to serialise it as bytes in memory first before writing out.