as-json icon indicating copy to clipboard operation
as-json copied to clipboard

Memory Allocation Failure When Handling Large JSON with Wazero (32 MB Limit)

Open snaffi opened this issue 1 month ago • 8 comments

Hello, My WASM scripts handle large JSON data—often several megabytes—and produce even larger JSON files.

I'm using wazero as the runtime, but it's limited to 32 MB. I’ve encountered a problem where the serializer fails because it can’t allocate enough memory. The only memory limit that works for me is 256 MB, but that’s too large—allocating ~256 MB for each operation is not ideal.

Input JSON size 2,195,285 2.19.mb Output JSON size 11,535,277 11.53mb

wasm error: unreachable
wasm stack trace:
	.~lib/rt/tlsf/allocateBlock(i32,i32) i32
	.~lib/rt/itcms/__new(i32,i32) i32
	.~lib/rt/itcms/__renew(i32,i32) i32
	.~lib/json-as/assembly/custom/sink/Sink#ensureCapacity(i32,i32)
	.~lib/json-as/assembly/custom/sink/Sink#write(i32,i32,i32,i32)
	.~lib/json-as/assembly/custom/sink/Sink#write@varargs(i32,i32,i32)
	.assembly/transformers/food/menu/body_output/FoodMenuBodyOutput#__SERIALIZE(i32) i32

snaffi avatar Jan 08 '25 07:01 snaffi