Memory Allocation Failure When Handling Large JSON with Wazero (32 MB Limit)

[snaffi]

Hello, My WASM scripts handle large JSON data—often several megabytes—and produce even larger JSON files.

I'm using wazero as the runtime, but it's limited to 32 MB. I’ve encountered a problem where the serializer fails because it can’t allocate enough memory. The only memory limit that works for me is 256 MB, but that’s too large—allocating ~256 MB for each operation is not ideal.

Input JSON size 2,195,285 2.19.mb Output JSON size 11,535,277 11.53mb

wasm error: unreachable
wasm stack trace:
	.~lib/rt/tlsf/allocateBlock(i32,i32) i32
	.~lib/rt/itcms/__new(i32,i32) i32
	.~lib/rt/itcms/__renew(i32,i32) i32
	.~lib/json-as/assembly/custom/sink/Sink#ensureCapacity(i32,i32)
	.~lib/json-as/assembly/custom/sink/Sink#write(i32,i32,i32,i32)
	.~lib/json-as/assembly/custom/sink/Sink#write@varargs(i32,i32,i32)
	.assembly/transformers/food/menu/body_output/FoodMenuBodyOutput#__SERIALIZE(i32) i32

[mattjohnsonpint]

My understanding is that Wazero allocates memory in 64 KB pages, growing up to the 4 GB max allowed in a single WASM memory - unless constraint by WithMemoryLimitPages.

https://github.com/tetratelabs/wazero/blob/610c202ec48f3a7c729f2bf11707330127ab3689/config.go#L57-L67

Where did you read that it is limited to 32 MB?

[snaffi]

I set that limit in code explicitly "WithMemoryLimitPages"

[JairusSW]

@snaffi, I'll take a look in a moment. Do you have a public repo where I can reproduce? If not I can just do it locally

[snaffi]

Unfortunately it's in private repo 😖

Code is also complex ... I'll try to provide simplified version by the end of the day

[JairusSW]

Thanks man 👍🏻

[snaffi]

Hello @JairusSW I've just got permission from company to share it with you

output JSON food-menu-body.out-2.json

output models Archive.zip

[JairusSW]

@snaffi, thanks. I'll take a look as soon as I'm out of classes

[JairusSW]

It's fixed as of v1.0.0-alpha.2. Note that it is alpha, so you can expect bugs until the 1.0.0 release becomes stable