Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)

[Fvoiretryzig]

Describe the bug

When I new an instance many times in go, it will show the error message after I execute instance, err := wasmer.NewInstance(module, importObject)

this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)

Steps to reproduce

1. This is my code,

package main

import (
	"fmt"
	wasmer "github.com/wasmerio/wasmer-go/wasmer"
	"io/ioutil"
)
var (
    module *wasmer.Module
    store *wasmer.Store
)
func run() {
    // Instantiates the module
    wasiEnvBuilder := wasmer.NewWasiStateBuilder("hello.wasm").CaptureStdout().CaptureStderr()
    wasiEnv, err := wasiEnvBuilder.Finalize()
    if err != nil {
        fmt.Println("this is wasiEnvBuilder.Finalize err!")
        fmt.Println(err)
        return
    }
    importObject, err := wasiEnv.GenerateImportObject(store, module)
    if err != nil {
        fmt.Println("this is wasiEnv.GenerateImportObject err!")
        fmt.Println(err)
        return
    }
    instance, err := wasmer.NewInstance(module, importObject)
	if err!=nil {
		fmt.Println("this is instance err:", err)
		return
	}
   start, err := instance.Exports.GetWasiStartFunction()
    if err != nil {
        fmt.Println("this is instance.Exports.GetWasiStartFunction err!")
        fmt.Println(err)
        return
    }
    start()
    fmt.Println(string(wasiEnv.ReadStdout()))
    instance.Close()
}
func main() {
	wasmBytes, _ := ioutil.ReadFile("hello.wasm")
	store = wasmer.NewStore(wasmer.NewEngine())
	module, _ = wasmer.NewModule(store, wasmBytes)
	for i:=0; i<100; i++ {
		fmt.Printf("this is %d times run\n", i)
		run()
	}
}

2. The wasm file is from hello.wasm in wasmer repository.
3. Then I run with "go run hello.go", it show these error message.

...
this is 83 times run
Hello, world!

this is 84 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 85 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 86 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 87 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 88 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 89 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 90 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 91 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 92 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 93 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 94 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 95 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 96 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 97 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 98 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)
this is 99 times run
this is instance err: Insufficient resources: Failed to create memory: Error when allocating memory: Cannot allocate memory (os error 12)

Expected behavior

I expect every run can output "hello world".

Actual behavior

It shows that in 84 times run it cannot new an instance correctly. A clear and concise description of what actually happened.

Additional context

I modify the example wasi.rs in wasmer repository to test if this case can run correctly in rust. This is the code I modified,

...
for i in 1..100000 {
    println!("this is {} times run", i);
    println!("Creating `WasiEnv`...");
    // First, we create the `WasiEnv`
    let mut wasi_env = WasiState::new("hello")
        // .args(&["world"])
        // .env("KEY", "Value")
        .finalize()?;

    println!("Instantiating module with WASI imports...");
    // Then, we get the import object related to our WASI
    // and attach it to the Wasm instance.
    let import_object = wasi_env.import_object(&module)?;
    let instance = Instance::new(&module, &import_object)?;

    println!("Call WASI `_start` function...");
    // And we just call the `_start` function!
    let start = instance.exports.get_function("_start")?;
    start.call(&[])?;
}
...

It can run correctly.

And I test another example in go. The wasm file I use is simple.wasm. I abandon the wasiEnv part, just like the code in simple.go. I add multiple times loop and add instance.Close() after every run. This can also run successfully.

I guess this maybe some problem with the implementation of wasi part in golang? How can I deal with this bug?

The libwasmer.so I used is the latest version v2.2.0(I just cloned from the wasmer repository master branch today.) My system version is as follows,

Linux k8snode01 4.9.253-tegra wasmerio/wasmer#1 SMP PREEMPT Mon Jul 26 12:13:06 PDT 2021 aarch64 aarch64 aarch64 GNU/Linux

Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic

              total        used        free      shared  buff/cache   available
Mem:        4059240      645880     1043860       29280     2369500     3198392
Swap:       2029616           0     2029616

And my go version is go version go1.16.13 linux/arm64.

[heyjdp]

Thanks for the awesome bug report. We are trying to reproduce this issue in Rust to understand the source of the issue.

[tglane]

I have encountered the same issue when using the rust implementation of wasmer 2.3.0 (and wasmer_wasi crate 2.3.0). I can instantiate 83 modules but the 84th one always errors with the same error message described by @Fvoiretryzig

Interestingly the error only occurs on my Raspberry Pi 4 with 4 Gb of Ram. On an x86 laptop with 8 or 16 Gb of Ram the error was not reproducible.

In this example I tried to start 200 modules using the same engine but a separate store for every module.

The error occurs in this function at line 6 where the instance is created.

If more information are needed feel free to ask. I hope this helps a little bit :)

[thedavidmeister]

kk so this is actually really bad, it's kind of a blocker for adoption in a system that is long running and creates many instances over time

i'm seeing this while in the process of migrating from 2.x to 4.x (which is weird because i did not see the problem in 2.x which is where the issue was originally reported)

https://github.com/holochain/holochain-wasmer/actions/runs/6482486633/job/17602102476?pr=99

it shows up in the benchmarks after building 21844 instances exactly, i see the same number on the CI as on my laptop

i do not see any spike in memory usage, as monitored by running htop alongside the benchmarks, and it seems to come from creating instances specifically, not modules

actually i have a cached module and am reusing it to build instances, i see cache hits on the module when i log it

similar but slightly different numbers by OS e.g. mac at 21837

https://github.com/holochain/holochain-wasmer/actions/runs/6482486622/job/17602102485?pr=99

[theduke]

This is not a bug, this is expected behaviour.

Wasmer needs to reserve about 4GB of memory space per instance (32 bit), plus a 2GB guard page. So that's roughly 6GB per instance.

This is required to ensure the memory isolation guarantees of Webassembly, while still providing reasonably fast memory access.

On Linux a userspace process can use about 2^47 bytes of memory. If we consider that the Rust and all the required plumbing also need space, the process will run out of available address space at about 20k+ instances.

If you want to go beyond that, you could experiment with the jsc (JavascriptCore) backend, which might use less memory, but I'm not familiar with the details of how JSC manages memory.

[theduke]

Just an addendum:

the wasmer-wasix crate has an async_threading feature, which makes it possible to pause instances persist, their state, and resume them later.

With some modifications it would be possible to persist the memory to disk and load again from disk later when resuming, which would free up address space.

[theduke]

And a second addendum: I just leanred that there is actually a different compilation mode where guard pages are not required, and where each memory access is bounds checked, which enables launching more instances with dynamically growing memory.

[thedavidmeister]

@theduke please don't close this issue until i've had time to respond/investigate

• i don't see memory being consumed in htop, is that expected?
• unless i have a bug (entirely possible) the benchmark that triggers the error is creating and dropping instances in a tight loop, so why does that cause memory to fill?
• could you link me to the docs on this alternate compilation mode so i can test it?
• wasix is indeed a nice thing that i'd like to try out, but first i need to finish my upgrade to 4.x :)

[theduke]

i don't see memory being consumed in htop, is that expected?

Yeah, that's expected. Memory isn't actually consumed from the OS perspective until it is actively used, but the runtime still needs to reserve that memory.

[theduke]

unless i have a bug (entirely possible) the benchmark that triggers the error is creating and dropping instances in a tight loop, so why does that cause memory to fill?

Could you provide a test case?

[thedavidmeister]

@theduke i've since refactored everything to avoid this, you can close the issue

[theduke]

@thedavidmeister great to hear.