wasm-micro-runtime
wasm-micro-runtime copied to clipboard
AOT compiled WASM couldn't be loaded on macOS M1 environment
Step to replicates
- Build ./sample/basic WASM application
- Move to wamr-compiler and build wamrc
- Built AOT object from ./out/wasm-apps/testapp.wasm with wamrc:
/path/to/wamr-compiler/build/wamrc --cpu=apple-m1 --size-level=3 -o ./out/wasm-apps/testapp.aot ./out/wasm-apps/testapp.wasm
Create AoT compiler with:
target: arm64
target cpu: apple-m1
cpu features:
opt level: 3
size level: 3
output format: AoT file
Compile success, file ./out/wasm-apps/testapp.aot was generated.
- Execute basic sample application with AOT object.
- Then, :boom:!
$ ./out/basic -f out/wasm-apps/testapp.aot
Load wasm module failed. error: AOT module load failed: mmap memory failed
Environment
Software versions:
$ sw_vers
ProductName: macOS
ProductVersion: 12.5
BuildVersion: 21G72
Host Compiler version:
$ clang --version
Apple clang version 13.1.6 (clang-1316.0.21.2.5)
Target: arm64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
WASI SDK's compiler version:
$ ~/work/wasi-sdk-16.0/bin/clang --version
clang version 14.0.4 (https://github.com/llvm/llvm-project 29f1039a7285a5c3a9c353d054140bf2556d4c4d)
Target: wasm32-unknown-wasi
Thread model: posix
InstalledDir: ~/work/wasi-sdk-16.0/bin
I tried to take a look on M1 Mac issue and I found the several issues before calling WASM function part:
- With macOS arm64 runtime, it is always hardened. We shouldn't use
PROT_EXEC
flags in mmap. Otherwise, mmap is failed.
diff --git a/core/shared/platform/common/posix/posix_memmap.c b/core/shared/platform/common/posix/posix_memmap.c
index 2dfbee4..1de331e 100644
--- a/core/shared/platform/common/posix/posix_memmap.c
+++ b/core/shared/platform/common/posix/posix_memmap.c
@@ -16,6 +16,12 @@ static size_t total_size_munmapped = 0;
#define HUGE_PAGE_SIZE (2 * 1024 * 1024)
+#if defined(__APPLE__) && defined(__arm64__)
+#define EXEC_FLAGS 0
+#else
+#define EXEC_FLAGS PROT_EXEC
+#endif
+
#if !defined(__APPLE__) && !defined(__NuttX__) && defined(MADV_HUGEPAGE)
static inline uintptr_t
round_up(uintptr_t v, uintptr_t b)
@@ -66,7 +72,7 @@ os_mmap(void *hint, size_t size, int prot, int flags)
map_prot |= PROT_WRITE;
if (prot & MMAP_PROT_EXEC)
- map_prot |= PROT_EXEC;
+ map_prot |= EXEC_FLAGS;
#if defined(BUILD_TARGET_X86_64) || defined(BUILD_TARGET_AMD_64)
#ifndef __APPLE__
@@ -243,7 +249,7 @@ os_mprotect(void *addr, size_t size, int prot)
map_prot |= PROT_WRITE;
if (prot & MMAP_PROT_EXEC)
- map_prot |= PROT_EXEC;
+ map_prot |= EXEC_FLAGS;
return mprotect(addr, request_size, map_prot);
}
- Default target name of M1 macOS is
arm64
notaarch64v8
diff --git a/core/iwasm/aot/arch/aot_reloc_aarch64.c b/core/iwasm/aot/arch/aot_reloc_aarch64.c
index 4c46eaa..dc3cb8f 100644
--- a/core/iwasm/aot/arch/aot_reloc_aarch64.c
+++ b/core/iwasm/aot/arch/aot_reloc_aarch64.c
@@ -53,7 +53,11 @@ get_target_symbol_map(uint32 *sym_num)
return target_sym_map;
}
+#if defined(__APPLE__) && defined(__arm64__)
+#define BUILD_TARGET_AARCH64_DEFAULT "arm64"
+#else
#define BUILD_TARGET_AARCH64_DEFAULT "aarch64v8"
+#endif
void
get_current_target(char *target_buf, uint32 target_buf_size)
{
With the above change, I still got bus error:
$ lldb -- ./out/basic -f out/wasm-apps/testapp.aot
(lldb) target create "./out/basic"
Current executable set to '/Users/cosmo/GitHub/wasm-micro-runtime/samples/basic/out/basic' (arm64).
(lldb) settings set -- target.run-args "-f" "out/wasm-apps/testapp.aot"
(lldb) r
Process 27342 launched: '/Users/cosmo/GitHub/wasm-micro-runtime/samples/basic/out/basic' (arm64)
Process 27342 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x1002dc004)
frame #0: 0x00000001002dc004
-> 0x1002dc004: str d8, [sp, #-0x70]!
0x1002dc008: stp x29, x30, [sp, #0x10]
0x1002dc00c: stp x28, x27, [sp, #0x20]
0x1002dc010: stp x26, x25, [sp, #0x30]
Target 0: (basic) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x1002dc004)
* frame #0: 0x00000001002dc004
frame #1: 0x000000010001c224 basic`call_func + 8
frame #2: 0x0000000100030fb0 basic`invoke_native_with_hw_bound_check + 444
frame #3: 0x0000000100030bc0 basic`aot_call_function + 420
frame #4: 0x00000001000199a4 basic`wasm_runtime_call_wasm_a + 740
frame #5: 0x0000000100001288 basic`main + 612
frame #6: 0x00000001000d508c dyld`start + 520
Hello, I'm getting the same error on MacOS M1, did you manage to find a solution?
No, I didn't. Currently, I disable to use AOT execution on M1 Mac.
@wenyongh Any hints on what changes would be needed to use AOT on M1 macs? After reading the docs, I was changing
diff --git a/core/iwasm/aot/aot_loader.c b/core/iwasm/aot/aot_loader.c
index b5fa33c0..854453ad 100644
--- a/core/iwasm/aot/aot_loader.c
+++ b/core/iwasm/aot/aot_loader.c
@@ -3232,6 +3232,7 @@ aot_load_from_aot_file(const uint8 *buf, uint32 size, char *error_buf,
if (!module)
return NULL;
+ pthread_jit_write_protect_np(false);
if (!load(buf, size, module, error_buf, error_buf_size)) {
aot_unload(module);
return NULL;
diff --git a/core/iwasm/aot/arch/aot_reloc_aarch64.c b/core/iwasm/aot/arch/aot_reloc_aarch64.c
index 4c46eaa0..10ec3f5a 100644
--- a/core/iwasm/aot/arch/aot_reloc_aarch64.c
+++ b/core/iwasm/aot/arch/aot_reloc_aarch64.c
@@ -53,7 +53,12 @@ get_target_symbol_map(uint32 *sym_num)
return target_sym_map;
}
+#if defined(__APPLE__) && defined(__arm64__)
+#define BUILD_TARGET_AARCH64_DEFAULT "arm64"
+#else
#define BUILD_TARGET_AARCH64_DEFAULT "aarch64v8"
+#endif
+
void
get_current_target(char *target_buf, uint32 target_buf_size)
{
diff --git a/core/shared/platform/common/posix/posix_memmap.c b/core/shared/platform/common/posix/posix_memmap.c
index 2dfbee45..9c403d66 100644
--- a/core/shared/platform/common/posix/posix_memmap.c
+++ b/core/shared/platform/common/posix/posix_memmap.c
@@ -36,7 +36,7 @@ void *
os_mmap(void *hint, size_t size, int prot, int flags)
{
int map_prot = PROT_NONE;
- int map_flags = MAP_ANONYMOUS | MAP_PRIVATE;
+ int map_flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_JIT;
uint64 request_size, page_size;
uint8 *addr = MAP_FAILED;
uint32 i;
And with that I managed fix some problems and get farther in the execution, but I still get
thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x100430008)
* frame #0: 0x0000000100430008
frame #1: 0x000000010006f5f0 iwasm`invoke_native_with_hw_bound_check(exec_env=0x0000000103600800, func_ptr=0x0000000100430008, func_type=0x0000000102400590, signature=0x0000000000000000, attachment=0x0000000000000000, argv=0x0000000000000000, argc=0, argv_ret=0x0000000000000000) at aot_runtime.c:1392:19
frame #2: 0x000000010006f160 iwasm`aot_call_function(exec_env=0x0000000103600800, function=0x000000016fdfdc80, argc=0, argv=0x0000000000000000) at aot_runtime.c:1601:15
frame #3: 0x000000010006e7b4 iwasm`execute_post_instantiate_functions(module_inst=0x0000000102703380, is_sub_inst=false, exec_env_main=0x0000000000000000) at aot_runtime.c:1022:14
frame #4: 0x000000010006d0b8 iwasm`aot_instantiate(module=0x0000000101ed0a40, parent=0x0000000000000000, exec_env_main=0x0000000000000000, stack_size=65536, heap_size=0, error_buf="", error_buf_size=128) at aot_runtime.c:1231:10
frame #5: 0x00000001000372a0 iwasm`wasm_runtime_instantiate_internal(module=0x0000000101ed0a40, parent=0x0000000000000000, exec_env_main=0x0000000000000000, stack_size=65536, heap_size=0, error_buf="", error_buf_size=128) at wasm_runtime_common.c:1340:44
frame #6: 0x0000000100037314 iwasm`wasm_runtime_instantiate(module=0x0000000101ed0a40, stack_size=65536, heap_size=0, error_buf="", error_buf_size=128) at wasm_runtime_common.c:1354:12
frame #7: 0x0000000100006d3c iwasm`main(argc=1, argv=0x000000016fdfecc8) at main.c:933:15
frame #8: 0x000000019a953f28 dyld`start + 2236
I tried adding the address sanitizer to get more info:
./iwasm -v=5 wasm-apps/no_pthread.aot
[5/5] Linking C executable iwasm
[10:45:10:477 - 1F610DB40]: Found malloc function, name: malloc, index: 17
[10:45:10:626 - 1F610DB40]: Found free function, name: free, index: 19
[10:45:10:632 - 1F610DB40]: Load module success.
[10:45:10:654 - 1F610DB40]: Memory instantiate:
[10:45:10:656 - 1F610DB40]: page bytes: 65536, init pages: 1, max pages: 30
[10:45:10:658 - 1F610DB40]: data offset: 5808, stack size: 32768
[10:45:10:660 - 1F610DB40]: heap offset: 65536, heap size: 0
AddressSanitizer:DEADLYSIGNAL
=================================================================
==37835==ERROR: AddressSanitizer: BUS on unknown address (pc 0x000100618008 bp 0x00016fc19800 sp 0x00016fc195d0 T0)
==37835==The signal is caused by a UNKNOWN memory access.
==37835==Hint: this fault was caused by a dereference of a high value address (see register values below). Disassemble the provided pc to learn which register was used.
#0 0x100618008 (<unknown module>)
#1 0x1002535ec in invoke_native_with_hw_bound_check aot_runtime.c:1392
#2 0x10025315c in aot_call_function aot_runtime.c:1601
#3 0x1002527b0 in execute_post_instantiate_functions aot_runtime.c:1022
#4 0x1002510b4 in aot_instantiate aot_runtime.c:1231
#5 0x10021b29c in wasm_runtime_instantiate_internal wasm_runtime_common.c:1340
#6 0x10021b310 in wasm_runtime_instantiate wasm_runtime_common.c:1354
#7 0x1001ead38 in main main.c:933
#8 0x19a953f24 (<unknown module>)
==37835==Register values:
x[0] = 0x0000000103600800 x[1] = 0x0000000000000000 x[2] = 0x0000000000000000 x[3] = 0x0000000000000000
x[4] = 0x0000000000000000 x[5] = 0x0000000000000000 x[6] = 0x0000000000000000 x[7] = 0x0000000000000000
x[8] = 0x0000000100221cf0 x[9] = 0x0000000000000000 x[10] = 0x0000000000000000 x[11] = 0x0000000000000001
x[12] = 0x0000000000000000 x[13] = 0xffffff8fd205cd1f x[14] = 0x0000000000007e01 x[15] = 0x0000000000000006
x[16] = 0x000000019acda3d0 x[17] = 0x00000001006b45e0 x[18] = 0x0000000000000000 x[19] = 0x0000000100618008
x[20] = 0x000000010021f1b0 x[21] = 0x0000000000000000 x[22] = 0x000000016fc195d0 x[23] = 0x000000019a9ce366
x[24] = 0x000000016fc1a9d0 x[25] = 0x0000000000000001 x[26] = 0x0000000000000000 x[27] = 0x0000000000000000
x[28] = 0x0000000000000000 fp = 0x000000016fc19800 lr = 0x0000000100221d74 sp = 0x000000016fc195d0
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: BUS (<unknown module>)
==37835==ABORTING
[1] 37835 abort ./iwasm -v=5 wasm-apps/no_pthread.aot
I don't know much about AOT, so I would appreciate any inputs or suggestions on what to try next.
Managed to get it to work, opening a PR
Seems to be resolved in https://github.com/bytecodealliance/wasm-micro-runtime/pull/2618.