Ranging over a large array makes the compiler eat my ram

[Jille]

// tinygo build -o repro.wasm -target wasi repro.go

package main

func main() {
	var buckets [1 << 24]int16
	for _, b := range buckets {
		_ = b
	}
}

tinygo starts consuming all my RAM.

Stacktrace of the compiler:

goroutine 75 [syscall]:
tinygo.org/x/go-llvm._Cfunc_LLVMRunFunctionPassManager(0x60000021c3a0, 0x600002c04208)
	_cgo_gotypes.go:8863 +0x4c
tinygo.org/x/go-llvm.PassManager.RunFunc.func1({0x4225af9?}, {0x2?})
	/Users/runner/go/pkg/mod/tinygo.org/x/[email protected]/ir.go:2010 +0x70
tinygo.org/x/go-llvm.PassManager.RunFunc({0x600003500580?}, {0xc000143cf0?})
	/Users/runner/go/pkg/mod/tinygo.org/x/[email protected]/ir.go:2010 +0x19
github.com/tinygo-org/tinygo/builder.Build.func1(0x0)
	/Users/runner/work/tinygo/tinygo/builder/build.go:393 +0xc2b
github.com/tinygo-org/tinygo/builder.runJob(0xc00010f1a0, 0x0?)
	/Users/runner/work/tinygo/tinygo/builder/jobs.go:222 +0x4f
created by github.com/tinygo-org/tinygo/builder.runJobs
	/Users/runner/work/tinygo/tinygo/builder/jobs.go:123 +0x5db

tinygo version 0.23.0 darwin/amd64 (using go version go1.18 and LLVM version 14.0.0)
Darwin 21.3.0 Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T6000 arm64

[dgryski]

Ranging over an addressable array as you are doing here causes Go to have to copy the array. You can prevent this by instead ranging over a slice:

	for _, b := range buckets[:] {

This is required by the specification. Upstream Go also has this "problem" but it is effectively invisible due to the better garbage collector.

[deadprogram]

Thank you for the excellent explanation @dgryski

Now closing as answered, please reopen if needed.

[Jille]

Agreed that ranging over the slice is the easy fix.

However, I'd still argue that the compiler taking 30GB+ RAM is a bug.

[aykevl]

I agree this is a bug and should be fixed. When we looked at the problem it appears that a LLVM pass is going wrong, it somehow hangs and eats all RAM.

[dgryski]

Good point. Seems like there must be something about how we're generating the IR for this particular case that causes the memory bloat.