zephyr: unable to compile with `WAMR_BUILD_LIBC_WASI` flag.

[lucasAbadFr]

Hello, I'm currently experimenting with sockets in WAMR on Zephyr, and I'm encountering some challenges.

1. I've adapted a section of the Zephyr socket API.
2. I'm attempting to run an HTTP client on WAMR (largely inspired by this sample).
3. I'm working with the stm32 nucleo-h563zi board, but I'm facing compilation issues.

Compilation issues

1. Plain compilation

   I compilated the http_client to a wasm module and like with the simple sample, I included it in a header file. But when I ran it on the board I had:

   *** Booting Zephyr OS build v3.6.0-961-gddb147d0a45d ***
   [00:00:00:000 - 20001190]: warning: failed to link import function (env, getaddrinfo)
   [00:00:00:000 - 20001190]: warning: failed to link import function (env, socket)
   [00:00:00:000 - 20001190]: warning: failed to link import function (env, connect)
   [00:00:00:000 - 20001190]: warning: failed to link import function (env, send)
   [00:00:00:000 - 20001190]: warning: failed to link import function (env, recv)
   [00:00:00:000 - 20001190]: warning: failed to link import function (env, close)
   [00:00:00:000 - 20001190]: warning: failed to link import function (env, __errno_location)
   Preparing HTTP GET request for http://192.0.2.10:8000/
   Exception: failed to call unlinked import function (env, getaddrinfo)
   elapsed: 72
   

   📄 Notes: I later adapted the sample to not use getaddrinfo because it add a useless dependency to netdb.h that I did not want.

2. WAMR_BUILD_LIBC_WASI flag

   After looking at the socket-api sample, I found that I need to use the WAMR_BUILD_LIBC_WASI flag, so I added this flag to the CMakeLists.txt file but it didn't work.

   The output of the compilation is the same described in this issue My errors concist mainly of:

   1. libc_errno.h not found.

      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/random.c:16:10: fatal error: libc_errno.h: No such file or directory
      16 | #include "libc_errno.h"
            |          ^~~~~~~~~~~~~~
      compilation terminated.
      

   2. nfds_t unknown type.

      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/blocking_op.h:57:65: error: unknown type name 'nfds_t'
      57 | blocking_op_poll(wasm_exec_env_t exec_env, struct pollfd *pfds, nfds_t nfds,
      

   3. pthread_condattr_t unknown type & CLOCK_MONOTONIC undeclared.

      # Skip previous logs
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:137:5: error: unknown type name 'pthread_condattr_t'
      137 |     pthread_condattr_t attr;
            |     ^~~~~~~~~~~~~~~~~~
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:139:9: warning: implicit declaration of function 'pthread_condattr_init' [-Wimplicit-function-declaration]
      139 |     if (pthread_condattr_init(&attr) != 0)
            |         ^~~~~~~~~~~~~~~~~~~~~
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:142:9: warning: implicit declaration of function 'pthread_condattr_setclock' [-Wimplicit-function-declaration]
      142 |     if (pthread_condattr_setclock(&attr, CLOCK_MONOTONIC) != 0)
            |         ^~~~~~~~~~~~~~~~~~~~~~~~~
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:142:42: error: 'CLOCK_MONOTONIC' undeclared (first use in this function)
      142 |     if (pthread_condattr_setclock(&attr, CLOCK_MONOTONIC) != 0)
            |                                          ^~~~~~~~~~~~~~~
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:142:42: note: each undeclared identifier is reported only once for each function it appears in
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:145:9: warning: implicit declaration of function 'pthread_cond_init'; did you mean 'os_thread_env_init'? [-Wimplicit-function-declaration]
      145 |     if (pthread_cond_init(&cond->object, &attr) != 0)
            |         ^~~~~~~~~~~~~~~~~
            |         os_thread_env_init
      /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:150:5: warning: implicit declaration of function 'pthread_condattr_destroy' [-Wimplicit-function-declaration]
      150 |     pthread_condattr_destroy(&attr);
            |     ^~~~~~~~~~~~~~~~~~~~~~~~
      # Skip following logs
      

3. Try to resolve

   I browsed the issues and found this Pull Request, but applying the fix lead to more errors and just one error was fixed.

   In core/shared/platform/zephyr/shared_platform.cmake I added the following lines:

   + if (WAMR_BUILD_LIBC_WASI EQUAL 1)
   +   list(APPEND source_all ${PLATFORM_SHARED_DIR}/../common/posix/posix_file.c)
   +   include (${CMAKE_CURRENT_LIST_DIR}/../common/libc-util/platform_common_libc_util.cmake)
   +   set(source_all ${source_all} ${PLATFORM_COMMON_LIBC_UTIL_SOURCE})
   + endif ()
   

   The error libc_errno.h was fixed but the other errors still persist. I also get all the errors from the /core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c file, mostly linked to the File System API ( e.g errors on struct pollfd).

4. Test on host

   I reused the socket-api sample to compile and run the http_client module and it worked on my host machine.

   ./iwasm --addr-pool=192.0.2.10 http_get.wasm
   Preparing HTTP GET request for http://127.0.0.1:8000/
   sock = 5
   Response:
   
   HTTP/1.0 200 OK
   Server: SimpleHTTP/0.6 Python/3.10.12
   Date: Thu, 11 Apr 2024 13:41:11 GMT
   Content-type: text/html
   Content-Length: 14054
   Last-Modified: Thu, 30 Mar 2023 09:11:09 GMT
   
   # Skip HTML content
   

Runtime configuration

1. Am I missing something ?

   My runtime configuration is the following:

   -- Build Configurations:
         Build as target THUMBV8
         CMAKE_BUILD_TYPE
         WAMR Interpreter enabled
         WAMR AOT enabled
         WAMR Fast JIT disabled
         WAMR LLVM ORC JIT disabled
         Libc builtin enabled
         Libc WASI enabled
         Fast interpreter disabled
         Multiple modules disabled
         Bulk memory feature enabled
         Wakeup of blocking operations enabled
         Reference types disabled
         GC performance profiling disabled
         Global heap pool enabled
         Custom global heap size: 131072
         Module instance context enabled
         Quick AOT/JIT entries enabled
   

2. How to use the address pool ?

   I would like to have some help with the libc_wasi_init(wasm_module, argc, argv, &wasi_parse_ctx) function because if I understood it's mandatory to run a network application on WAMR.

   I looked at the product-mini/plateforms/posix/main.c & product-mini/plateforms/common/libc_wasi.c files but didn't fully understand the implementation because it is made to parse arguments from the command line.

   Meanwhile, I'm trying to run the http_client module on the board, so I don't have the possibility to pass arguments to the module.

   I came up with this configuration before wasm_runtime_full_init(&init_args) call in main.c.

   #if WASM_ENABLE_LIBC_WASI != 0
   #define HUMAN_READABLE_ADDRESS "192.0.2.10\\24"
   libc_wasi_parse_context_t wasi_parse_ctx;
   memset(&wasi_parse_ctx, 0, sizeof(wasi_parse_ctx));
   
   libc_wasi_parse_result_t result = libc_wasi_parse(HUMAN_READABLE_ADDRESS, &wasi_parse_ctx);
   switch (result) {
         case LIBC_WASI_PARSE_RESULT_OK:
               continue;
         case LIBC_WASI_PARSE_RESULT_NEED_HELP:
               return;
         case LIBC_WASI_PARSE_RESULT_BAD_PARAM:
               return;
   }
   libc_wasi_init(wasm_module, argc, argv, &wasi_parse_ctx);
   #endif
   

   But I'm not sure if it's the right way to use the libc_wasi_parse function.

3. Should I change build policy ?

   I tried to adapt the CMakeLists.txt file from the simple sample. What I changed:

   1. I added the WAMR_BUILD_LIBC_WASI flag.
   2. I added some env conf (path to wamr-sdk, wasi-sdk, python, ...)
   3. I added the wamr-sdk as an external projecxt and compile the libapp_framework.a.
   4. I added a custom target to compile the http_client module.
   5. I added a custom command that use a python script to generate the http_client.h module.
   • .wasm size ~= 1.3kB
   • .h size ~= 7.6kB

Any help will be appreciated. Thanks.

[lucasAbadFr]

Status

1. Abstracting libc primitives: After looking at how the POSIX primitives where abstracted in core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/ to support Windows. I think that we could do the same to support Zephyr, with the final idea to have an implementation entirely relying on the os_ APIs.

   • functions:
     • poll --> os_poll
     • ioctl --> os_ioctl
   • structs/types:
     • timespec --> os_timespec
     • pollfd --> os_poll_file_handle
     • nfds_t --> os_nfds_t

2. Extending platform_api_extension.h: The following API were added.

   __wasi_errno_t
   os_ioctl(os_file_handle *handle, int request, void *argp);
   
   // Not sure for return type
   __wasi_errno_t
   os_poll(os_poll_file_handle *pfds, os_nfds_t nfs, int timeout);
   

3. Changing the sandboxed system primitives to use the new types and functions.

   • random.c:

     #elif defined(BH_PLATFORM_ZEPHYR)
     static void
     open_urandom(void)
     {
         // Not implemented
     }
     __wasi_errno_t
     random_buf(void *buf, size_t len)
     {
         return __WASI_ENOSYS;
     }
     

   • blocking.h:

     __wasi_errno_t
     - os_poll(pollfd *pfds, os_nfds_t nfds, int timeout)
     + os_poll(os_poll_file_handle *pfds, os_nfds_t nfds, int timeout)
     

   • locking.h: change every struct timespec to struct os_timespec
   • posix.c: change every struct pollfd to struct os_poll_file_handle

4. Defining new types in platform_internal.h: Based on the os_file_handle type implementation we define the following types (and flags).

   typedef struct zsock_pollfd os_poll_file_handle;
   typedef unsigned int os_nfds_t;
   
   // To use posix flags
   #define POLLIN ZSOCK_POLLIN
   #define POLLPRI ZSOCK_POLLPRI
   #define POLLOUT ZSOCK_POLLOUT
   #define POLLERR ZSOCK_POLLERR
   #define POLLHUP ZSOCK_POLLHUP
   #define POLLNVAL ZSOCK_POLLNVAL
   
   #define FIONREAD ZFD_IOCTL_FIONREAD
   
   typedef struct {
           time_t tv_sec;
           long tv_nsec;
   } os_timespec;
   
   #define CLOCK_REALTIME 1
   #define CLOCK_MONOTONIC 4
   
   typedef struct {
       struct k_mutex mtx;  // Mutex for exclusive access
       struct k_sem sem;    // Semaphore for shared access
       int read_count;      // Number of readers
   } korp_rwlock;
   

5. Implementing the new functions in platform_socket.c:

   __wasi_errno_t
   os_ioctl(os_file_handle *handle, int request, void *argp)
   {
       __wasi_errno_t wasi_errno = __WASI_ESUCCESS;
   
       if(zsock_ioctl_wrapper(handle, request, argp) < 0){
           wasi_errno = zephyr_to_wasi_errno(errno);
       }
       return wasi_errno;
   }
   
   __wasi_errno_t
   os_poll(os_poll_file_handle *fds, os_nfds_t nfs, int timeout)
   {
       __wasi_errno_t wasi_errno = __WASI_ESUCCESS;
       int rc = 0;
   
       rc = zsock_poll(fds, nfs, timeout) 
       if(rc < 0){
           wasi_errno = zephyr_to_wasi_errno(errno);
       }
       switch(rc){
           case 0:
               wasi_errno = __WASI_ETIMEOUT;
               break;
           case -1:
               wasi_errno = zephyr_to_wasi_errno(errno);
               break;
           default:
               break;
       }
       return wasi_errno;
   }
   

6. compilation errors The compilation still fail with the same warnings as before. The unknown flags errors have disappeared. But the warning about the struct timespec and struct pollfd still persist.

   • Error on locking.h:

     /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:200:12: error: variable 'ts' has initializer but incomplete type
     200 |     struct os_timespec ts = {
         |            ^~~~~~~~~~~
     /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:201:10: error: 'struct os_timespec' has no member named 'tv_sec'
     201 |         .tv_sec = (time_t)(timeout / 1000000000),
         |          ^~~~~~
     /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:201:19: warning: excess elements in struct initializer
     201 |         .tv_sec = (time_t)(timeout / 1000000000),
         |                   ^
     /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/locking.h:201:19: note: (near initialization for 'ts')
     

   • Error on posix.c:

     /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:2153:61: error: invalid application of 'sizeof' to incomplete type 'struct os_poll_file_handle'
     2153 |         wasm_runtime_malloc((uint32)(nsubscriptions * sizeof(*pfds)));
         |                                                             ^
     /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:2177:25: error: invalid use of undefined type 'struct os_poll_file_handle'
     2177 |                     pfds[i] = (struct os_poll_file_handle){
         |                         ^
     /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:2178:26: error: 'struct os_poll_file_handle' has no member named 'fd'
     2178 |                         .fd = fos[i]->file_handle,
         |                          ^~
     /home/user/wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:2178:31: warning: excess elements in struct initializer
     2178 |                         .fd = fos[i]->file_handle,
         |
     

   This look like the compiler is not able to find the definition of the new types.

   • I tried to include the platform_internal.h in the locking.h and posix.c files to see if it solves the issue. But it didn't work.
   • I also tried to directly define the types in a new header and directly include it in the locking.h and posix.c files. But the issue still persist.
   • I even defined the whole file System API in a zephyr_file.c file (just returning __WASI_ENOSYS for now), and changed shared_platform.cmake. But the issue still persist.

   I'm still stuck on this issue. I will try to find a solution.

   If anyone is familiar with the implemenation of the sanboxed system primitives, or the how the abstract POSIX APIs are passed to sanboxed system primitives.

   Any help on this would be appreciated.

[wenyongh]

@lucasAbadFr could you upload the patch or create a PR and change it to draft or WIP, so that others can have a try according to your work?

[lucasAbadFr]

@wenyongh thanks for your reply.

A new zephyr sample called simpe-http was added to demonstrate socket usage on zephyr. Refer to the sample readme for build purpose.

I've just opened a draft and noticed that I didn't adhere to the code guideline or other guidelines.

These changes will cause most tests to fail.

Please note that the work wasn't intended to be shared in this state.

[lucasAbadFr]

Hi @wenyongh,

I managed to make a simple wasi module work (by breaking a lot of things).

I would be interested to know if it work on someone else board.

I updated the readme.md under the new sample to give a brief overview of what was done.

Also i'm open to any tips to compile a WASI module with libc (from wasi-sdk) linked.

[wenyongh]

@lucasAbadFr I am not sure who is using zephyr but I believe there are some developers requiring libc-wasi on zephyr. I found there are many modifications in wasm_runtime_common.c and core/iwasm/libraries/libc-wasi and wonder whether it is necessary? It would be better to add APIs and structures in zephyr platform and leave some APIs empty (or return false), and reduce the modification on core/iwasm and make the CIs run successfully.

[lucasAbadFr]

Thanks for the feedback.

The CI fail due to the new struct and API defined in platform_api_extension.h, to make it pass I could just typedef the new struct in platform_internal.h, and declare the API in their respective folder.

As for the changes in core/iwasm/libraries/libc-wasi/sandboxed-system-primitives, they are mostly here to abstract POSIX functions and use os_ abstraction. I think this is necessary to have at least a full abstraction.

The main changes are in core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c where I had an offset due to struct size being different on both platform.

I'm curently investigating how to solve this issue and working to maintain compatibility with POSIX platform, which is the most used/maintained one.

[wenyongh]

OK, I read the code and added some comments, it should be OK to abstract some os_xxx structures and functions. I think maybe you can enable the basic functionality and make CI run fine first, and then we can let others help review this PR and give suggestions, like how to abstract the APIs.