AOT invoking import without attachment

[bnason-nf]

Hi,

I have a workload that works fine with a WASM file, but when compiled to AOT, it fails. The difference in behavior seems to happen when the embedding application calls a function in the AOT module named __wasm_call_ctors(). The implementation of this function in turn calls back to native code through an import named _environ_sizes_get. This seems to be working, except in the AOT case, the attachment is empty.

This workload is very large, and so far I haven't been able to find a smaller repro case, so I'd like to investigate more myself to debug it. Can you please help me understand how this works? In this case where an AOT function calls an import, where is the mechanism that should make sure that the attachment is set before invoking the import? I see it for interpreted WASM, but for AOT I'm having trouble finding it.

Thanks, Benbuck

[yamt]

what do you mean by "attachment"?

[bnason-nf]

Hi @yamt, an attachment is basically a per-function user data for native (import) functions/symbols:

https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/core/iwasm/include/lib_export.h#L20-L27

typedef struct NativeSymbol {
    const char *symbol;
    void *func_ptr;
    const char *signature;
    /* attachment which can be retrieved in native API by
       calling wasm_runtime_get_function_attachment(exec_env) */
    void *attachment;
} NativeSymbol;

https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/core/iwasm/include/wasm_export.h#L1557-L1565

/**
 * Get attachment of native function from execution environment
 *
 * @param exec_env the execution environment to retrieve
 *
 * @return the attachment of native function
 */
WASM_RUNTIME_API_EXTERN void *
wasm_runtime_get_function_attachment(wasm_exec_env_t exec_env);

[yamt]

ah, that attachment. is your _environ_sizes_get a modified version of wasi's one? (just a guess from the name) i suspect the attachment is not implemented for the "call native func directly" case.

[bnason-nf]

Oh, that's an interesting idea. Is there a way to tell if that's what's happening? I know that it's calling my function, since the problem is that my function can't get the expected attachment. Is there some special logic somewhere that decides it doesn't need to set the attachment? If so, would you mind point that out please?

[bnason-nf]

@yamt are you talking about this line in libc_wasi_wrapper.c?

REG_NATIVE_FUNC(environ_sizes_get, "(**)i"),

If so, I tried removing that line, and also tried completely disabling libc WASI, and still have the same problem.

[bnason-nf]

Is call_aot_invoke_c_api_native() in aot_emit_function.c where the calling code is generated for this case? If so, what is the mechanism for setting the attachment?

[yamt]

Oh, that's an interesting idea. Is there a way to tell if that's what's happening? I know that it's calling my function, since the problem is that my function can't get the expected attachment. Is there some special logic somewhere that decides it doesn't need to set the attachment? If so, would you mind point that out please?

i meant this: https://github.com/bytecodealliance/wasm-micro-runtime/pull/3610 (not tested at all)

[bnason-nf]

Thanks @yamt, unfortunately that doesn't seem to change anything. Looking at that code, it seems like it needs to get the attachment from the import function (the same way it gets the signature), and then store that attachment in the exec_env, but it doesn't do that. Does something else do it?

[bnason-nf]

@yamt once I got the idea of what your #3610 change was intending to do, I was able to make a similar change that does fix the problem I'm seeing:

diff --git a/core/iwasm/compilation/aot_emit_function.c b/core/iwasm/compilation/aot_emit_function.c
index 8f6e3e45..40100a55 100644
--- a/core/iwasm/compilation/aot_emit_function.c
+++ b/core/iwasm/compilation/aot_emit_function.c
@@ -1412,7 +1412,7 @@ aot_compile_op_call(AOTCompContext *comp_ctx, AOTFuncContext *func_ctx,
     /* Get function type */
     if (func_idx < import_func_count) {
         func_type = import_funcs[func_idx].func_type;
-        signature = import_funcs[func_idx].signature;
+        // signature = import_funcs[func_idx].signature;
     }
     else {
         func_type =
diff --git a/core/iwasm/compilation/aot_llvm.c b/core/iwasm/compilation/aot_llvm.c
index df07c3ca..faa29994 100644
--- a/core/iwasm/compilation/aot_llvm.c
+++ b/core/iwasm/compilation/aot_llvm.c
@@ -2598,8 +2598,8 @@ aot_create_comp_context(const AOTCompData *comp_data, aot_comp_option_t option)
     if (option->enable_stack_estimation)
         comp_ctx->enable_stack_estimation = true;
 
-    if (option->quick_invoke_c_api_import)
-        comp_ctx->quick_invoke_c_api_import = true;
+    // if (option->quick_invoke_c_api_import)
+    //     comp_ctx->quick_invoke_c_api_import = true;
 
     if (option->llvm_passes)
         comp_ctx->llvm_passes = option->llvm_passes;

This change is clearly not what we want, but I believe it does show that the quick invoke path has an issue, it's not putting the attachment into exec_env.

[yamt]

how exactly did you add your _environ_sizes_get?

[bnason-nf]

I added it without a signature if that's what you mean.

[yamt]

• can you share your code around eg. EXPORT_WASM_API_WITH_ATT?
• are you using wasm_c_api.h at all?

[bnason-nf]

Hi @yamt, yes of course. At the lowest level, it's just a loop with this:

    NativeSymbol &nativeSymbol = nativeSymbols[s];
    nativeSymbol.symbol        = strdup(nativeFunction.name().c_str());
    nativeSymbol.func_ptr      = const_cast<void *>(nativeFunction.address());
    nativeSymbol.signature     = nullptr; // empty signature disables internal mismatch checks and conversions
    nativeSymbol.attachment    = nativeFunction.userData();

And then this array is then passed to wasm_runtime_register_natives() as you'd expect.

I'm not using wasm_c_api.h.

[yamt]

I'm not using wasm_c_api.h.

why are you using wamrc --invoke-c-api-import equivalent then?

[bnason-nf]

Hi @yamt, thanks for catching that, it was a mistake. However, after removing that, I still have the same problem.

[yamt]

i'm still a bit puzzled. https://github.com/bytecodealliance/wasm-micro-runtime/issues/3605#issuecomment-2221659635 implies that the function signature of your function (_environ_sizes_get) was available to wamrc. do you have any idea how it can happen? have you done anything to make wamrc know about your function?

Hi @yamt, thanks for catching that, it was a mistake. However, after removing that, I still have the same problem.

even with https://github.com/bytecodealliance/wasm-micro-runtime/pull/3610?

[wenyongh]

@bnason-nf Just as you said, the quick invoke path has an issue, it's not putting the attachment into exec_env, this is a limitation when directly calling the native API from aot code. But since by default the runtime builtin native APIs don't take attachment in them, it should not be an issue for these APIs. And I saw you set nativeSymbol.signature = nullptr;, so even if you use wamrc --native-lib=xxx.so to emit the aot code, wamrc won't emit optimized code to call native API directly. It seems that aot code calls aot_invoke_native/aot_call_indirect to invoke the native API, but I found that attachment is obtained in them and passed to wasm_runtime_invoke_native. A little strange, did you use wamrc --native-lib=xxx.so option?

[bnason-nf]

i'm still a bit puzzled. #3605 (comment) implies that the function signature of your function (_environ_sizes_get) was available to wamrc. do you have any idea how it can happen? have you done anything to make wamrc know about your function?

Yes, I'm confused too, I appreciate your help and patience.

Hi @yamt, thanks for catching that, it was a mistake. However, after removing that, I still have the same problem.

even with #3610?

My latest testing, without the --invoke-c-api-import option, without any changes to AOT compilation (such as #3610), and without the WASI equivalent functions that I was providing, does not run into the original problem I was reporting.

[bnason-nf]

@bnason-nf Just as you said, the quick invoke path has an issue, it's not putting the attachment into exec_env, this is a limitation when directly calling the native API from aot code. But since by default the runtime builtin native APIs don't take attachment in them, it should not be an issue for these APIs. And I saw you set nativeSymbol.signature = nullptr;, so even if you use wamrc --native-lib=xxx.so to emit the aot code, wamrc won't emit optimized code to call native API directly. It seems that aot code calls aot_invoke_native/aot_call_indirect to invoke the native API, but I found that attachment is obtained in them and passed to wasm_runtime_invoke_native. A little strange, did you use wamrc --native-lib=xxx.so option?

Thanks, I think I understand how this all works a little better now, and that I was doing unexpected things. Just to answer your question, I have not been using the --native-lib option at all.

[yamt]

My latest testing, without the --invoke-c-api-import option, without any changes to AOT compilation (such as #3610), and without the WASI equivalent functions that I was providing, does not run into the original problem I was reporting.

can you explain "without the WASI equivalent functions that I was providing"?

[bnason-nf]

can you explain "without the WASI equivalent functions that I was providing"?

From my application I was providing a handful of functions using wasm_runtime_register_natives(), such as environ_get_sizes, that are also provided by the WAMR libc WASI implementation in libc_wasi_wrapper.c. I wasn't aware that these were conflicting with each other and evidently causing problems.

[yamt]

can you explain "without the WASI equivalent functions that I was providing"?

From my application I was providing a handful of functions using wasm_runtime_register_natives(), such as environ_get_sizes, that are also provided by the WAMR libc WASI implementation in libc_wasi_wrapper.c. I wasn't aware that these were conflicting with each other and evidently causing problems.

you said your function's import name was "_environ_sizes_get". was it wrong?

[bnason-nf]

you said your function's import name was "_environ_sizes_get". was it wrong?

Yes that was confusing, sorry. The internal implementation name has the underscore, but the string passed to wasm_runtime_register_natives() was "environ_sizes_get" (without the underscore), the same as the libc WASI one:

    REG_NATIVE_FUNC(environ_sizes_get, "(**)i"),

[yamt]

if your function has the exactly same import module_name/name as wasi, it's the expected behavior. you can avoid it by building wamrc with -DWAMR_BUILD_LIBC_WASI=0.

[bnason-nf]

I just tried disabling libc WASI, and then providing my own WASI function equivalents again, and now I am getting the original missing attachment problem again.

[bnason-nf]

you said your function's import name was "_environ_sizes_get". was it wrong?

I just realized you were asking something different than I thought. Yes it was wrong, it was always sizes_get, not get_sizes. My first comment was correct, and the later one that reversed them was just a typo.

[yamt]

you said your function's import name was "_environ_sizes_get". was it wrong?

I just realized you were asking something different than I thought. Yes it was wrong, it was always sizes_get, not get_sizes. My first comment was correct, and the later one that reversed them was just a typo.

sizes_get vs get_sizes were all typo. my question was about the _ prefix.

[bnason-nf]

LOL, ok. Confusion on top of confusion.

[yamt]

I just tried disabling libc WASI, and then providing my own WASI function equivalents again, and now I am getting the original missing attachment problem again.

just to be sure, have you disabled wasi in wamrc?

[bnason-nf]

No, is there a way to do that? I meant that I disabled it in the runtime.