wasi-libc fails to build

[jmgurney]

When attempting to build wasm-libc, I get the following error:

$ make
[...]
diff -wur /Users/jmg/wasm/wasi-libc/expected/wasm32-wasi/defined-symbols.txt /Users/jmg/wasm/wasi-libc/sysroot/share/wasm32-wasi/defined-symbols.txt
--- /Users/jmg/wasm/wasi-libc/expected/wasm32-wasi/defined-symbols.txt	2022-12-20 11:19:59
+++ /Users/jmg/wasm/wasi-libc/sysroot/share/wasm32-wasi/defined-symbols.txt	2022-12-20 11:26:07
@@ -37,6 +37,7 @@
 __des_setkey
 __do_des
 __duplocale
+__emutls_v.errno
 __env_rm_add
 __errno_location
 __exp2f_data
make: *** [check-symbols] Error 1

I have simply cloned wasm-libc, and did a make in the directory. I am not sure why it is failing.

Environment: macOS Ventura 13.0.1, MBP 14-inch, 2021, M1 Pro, Rosetta 2 is NOT installed, so only native arm binaries.

$ make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0
$ clang --version
clang version 13.0.1
Target: arm64-apple-darwin22.1.0
Thread model: posix
InstalledDir: /opt/local/libexec/llvm-13/bin
$ llvm-ar --version
LLVM (http://llvm.org/):
  LLVM version 13.0.1
  Optimized build.
  Default target: arm64-apple-darwin22.1.0
  Host CPU: cyclone
$ llvm-nm --version
llvm-nm, compatible with GNU nm
LLVM (http://llvm.org/):
  LLVM version 13.0.1
  Optimized build.
  Default target: arm64-apple-darwin22.1.0
  Host CPU: cyclone
$ clang -print-targets
  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_32 - AArch64 (little endian ILP32)
    aarch64_be - AArch64 (big endian)
    amdgcn     - AMD GCN GPUs
    arm        - ARM
    arm64      - ARM64 (little endian)
    arm64_32   - ARM64 (little endian ILP32)
    armeb      - ARM (big endian)
    avr        - Atmel AVR Microcontroller
    bpf        - BPF (host endian)
    bpfeb      - BPF (big endian)
    bpfel      - BPF (little endian)
    hexagon    - Hexagon
    lanai      - Lanai
    mips       - MIPS (32-bit big endian)
    mips64     - MIPS (64-bit big endian)
    mips64el   - MIPS (64-bit little endian)
    mipsel     - MIPS (32-bit little endian)
    msp430     - MSP430 [experimental]
    nvptx      - NVIDIA PTX 32-bit
    nvptx64    - NVIDIA PTX 64-bit
    ppc32      - PowerPC 32
    ppc32le    - PowerPC 32 LE
    ppc64      - PowerPC 64
    ppc64le    - PowerPC 64 LE
    r600       - AMD GPUs HD2XXX-HD6XXX
    riscv32    - 32-bit RISC-V
    riscv64    - 64-bit RISC-V
    sparc      - Sparc
    sparcel    - Sparc LE
    sparcv9    - Sparc V9
    systemz    - SystemZ
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    wasm32     - WebAssembly 32-bit
    wasm64     - WebAssembly 64-bit
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64
    xcore      - XCore

Let me know if you need additional environment information.

Thanks.

[sbc100]

It could be that the expectations were built with llvm 14 and using llvm 13 can produce slightly different results. It looks like we test with 10 and 14: https://github.com/WebAssembly/wasi-libc/blob/957c7113c36c6d8c668fcf1d2bdd01b84e2e2530/.github/workflows/main.yml#L12

We do make some effort to try to account for different llvm version here: https://github.com/WebAssembly/wasi-libc/blob/957c7113c36c6d8c668fcf1d2bdd01b84e2e2530/Makefile#L621-L632, but perhaps we need to add something.

I think its safe to say is not consequential and you can probably ignore it.

[jmgurney]

So, just tried w/ clang 14 and get the same results (also tested w/ 15 and fails in the same way).

$ clang --version
clang version 14.0.6
Target: arm64-apple-darwin22.1.0
Thread model: posix
InstalledDir: /opt/local/libexec/llvm-14/bin
$ llvm-ar --version
LLVM (http://llvm.org/):
  LLVM version 14.0.6
  Optimized build.
  Default target: arm64-apple-darwin22.1.0
  Host CPU: cyclone
$ llvm-nm --version
llvm-nm, compatible with GNU nm
LLVM (http://llvm.org/):
  LLVM version 14.0.6
  Optimized build.
  Default target: arm64-apple-darwin22.1.0
  Host CPU: cyclone

[sunfishcode]

"__emutls_v" indicates that LLVM's "emulated TLS" feature is getting enabled. Could the Apple-distributed clang have something like -femulated-tls enabled by default? If that's true, I wonder if we could work around the problem by adding -fno-emulated-tls to the wasi-libc build.

[jmgurney]

Note that these are the MacPort installed versions of clang (installed in /opt), and not Apple's version.

Apple's version of clang does not have wasm support compiled in.

[jmgurney]

minor note, I did end up compiling wasi-sdk which compiles wasi-libc as part of it because the MacPort's clang is missing the file libclang_rt.builtins-wasm32.a which prevents anything from compiling. I was hoping to avoid having to do that as I already had clang/llvm compiled.

[sunfishcode]

Does that mean it's MacPorts that's enabling -femulated-tls? If so, then we may need a way to add -fno-emulated-tls in order to disable it.

[jmgurney]

Not sure. It looks like it's "enabled" for 10.6 and before: https://github.com/macports/macports-ports/blob/master/lang/llvm-14/Portfile

But that shouldn't made a difference, as I'm not on that version of MacOSX.

So I don't know how/why it's being enabled.

I also don't know HOW to enable emulated tls, so I don't really know where to look.

[GregoryConrad]

Not really sure if you're looking for a fix here or a workaround, but wasi-libc builds fine on my M1 Mac using homebrew's llvm (brew install llvm):

# Set up WASI Libc sysroot
LLVM_BIN="/opt/homebrew/opt/llvm/bin"
WASI_VERSION_TAG="wasi-sdk-19"
WASI_LIBC_ROOT=wasi-libc
git clone --recursive https://github.com/WebAssembly/wasi-libc $WASI_LIBC_ROOT
(
    cd $WASI_LIBC_ROOT &&
    git checkout $WASI_VERSION_TAG &&
    make CC=$LLVM_BIN/clang AR=$LLVM_BIN/llvm-ar NM=$LLVM_BIN/llvm-nm THREAD_MODEL=posix # TODO remove THREAD_MODEL?
)

[gian-didom]

On my Mac M1, the build fails. Even when supplying the llvm paths from brew, this is what I get:

In file included from dlmalloc/src/malloc.c:1468:

In file included from dlmalloc/src/malloc.c:1468:
sw/libs/wasi-libc/sysroot/include/stdlib.h:85:13: error: unknown type name 'wchar_t'
   85 | int mbtowc (wchar_t *__restrict, const char *__restrict, size_t);
      |             ^
sw/libs/wasi-libc/sysroot/include/stdlib.h:87:18: error: unknown type name 'wchar_t'
   87 | size_t mbstowcs (wchar_t *__restrict, const char *__restrict, size_t);
      |                  ^
sw/libs/wasi-libc/sysroot/include/stdlib.h:88:42: error: unknown type name 'wchar_t'
   88 | size_t wcstombs (char *__restrict, const wchar_t *__restrict, size_t);
      |                                          ^
3 errors generated.
make: *** [build/wasm32-wasi/dlmalloc/src/dlmalloc.o] Error 1

Any suggestion?

[sbc100]

Could you share the full failing command line along with the full error output? (Perhaps just re-run make and attache the entire output?)

[gian-didom]

This is the full output from make, along with system specifications:

MacBook Pro 16", 2021, Apple M1 Pro 16 GB OS: Sonoma 14.1

Flags have been set with:

  export LDFLAGS="-L/opt/homebrew/opt/llvm/lib"
  export CPPFLAGS="-I/opt/homebrew/opt/llvm/include"
  export LLVM_BIN="/opt/homebrew/opt/llvm/bin"

usr@MBPofUser wasi-libc %  make CC=$LLVM_BIN/clang AR=$LLVM_BIN/llvm-ar NM=$LLVM_BIN/llvm-nm
#
# Install the include files.
#
mkdir -p "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include"
cp -r "libc-bottom-half/headers/public"/* "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include"
# Generate musl's bits/alltypes.h header.
mkdir -p "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/bits"
sed -f libc-top-half/musl/tools/mkalltypes.sed \
            libc-top-half/musl/arch/wasm32/bits/alltypes.h.in \
            libc-top-half/musl/include/alltypes.h.in \
            > "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/bits/alltypes.h"
# Copy in the bulk of musl's public header files.
cp -r "libc-top-half/musl/include"/* "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include"
# Copy in the musl's "bits" header files.
cp -r "libc-top-half/musl"/arch/generic/bits/* "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/bits"
cp -r "libc-top-half/musl"/arch/wasm32/bits/* "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/bits"
# Remove selected header files.
rm -f /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/syscall.h.in" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/alltypes.h.in" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"alltypes.h.in" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"stdarg.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"stddef.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/errno.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/procfs.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/user.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/kd.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/vt.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/soundcard.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/sem.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/shm.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/msg.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/ipc.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/ptrace.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/statfs.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/kd.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/vt.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/soundcard.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/sem.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/shm.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/msg.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/ipc.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/ptrace.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/statfs.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/vfs.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/statvfs.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"syslog.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/syslog.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"wait.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/wait.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"ucontext.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/ucontext.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"paths.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"utmp.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"utmpx.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"lastlog.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/acct.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/cachectl.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/epoll.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/reboot.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/swap.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/sendfile.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/inotify.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/quota.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/klog.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/fsuid.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/io.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/prctl.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/mtio.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/mount.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/fanotify.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/personality.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"elf.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"link.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/link.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"scsi/scsi.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"scsi/scsi_ioctl.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"scsi/sg.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/auxv.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"pwd.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"shadow.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"grp.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"mntent.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"netdb.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"resolv.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"pty.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"dlfcn.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"setjmp.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"ulimit.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/xattr.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"wordexp.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"spawn.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/membarrier.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/signalfd.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"termios.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/termios.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"bits/termios.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"net/if.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"net/if_arp.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"net/ethernet.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"net/route.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"netinet/if_ether.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"netinet/ether.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/timerfd.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"libintl.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"sys/sysmacros.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"aio.h" /path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/"pthread.h"
clang -O2 -DNDEBUG --target=wasm32-wasi -fno-trapping-math -Wall -Wextra -Werror -Wno-null-pointer-arithmetic -Wno-unused-parameter -Wno-sign-compare -Wno-unused-variable -Wno-unused-function -Wno-ignored-attributes -Wno-missing-braces -Wno-ignored-pragmas -Wno-unused-but-set-variable -Wno-unknown-warning-option -mthread-model single -isystem "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include" -Ilibc-bottom-half/headers/private -Ilibc-bottom-half/cloudlibc/src/include -Ilibc-bottom-half/cloudlibc/src -Ilibc-top-half/musl/src/include -Ilibc-top-half/musl/src/internal -fPIC -fvisibility=default -MD -MP -o build/wasm32-wasi/libc-bottom-half/crt/crt1-command.o -c libc-bottom-half/crt/crt1-command.c
clang -O2 -DNDEBUG --target=wasm32-wasi -fno-trapping-math -Wall -Wextra -Werror -Wno-null-pointer-arithmetic -Wno-unused-parameter -Wno-sign-compare -Wno-unused-variable -Wno-unused-function -Wno-ignored-attributes -Wno-missing-braces -Wno-ignored-pragmas -Wno-unused-but-set-variable -Wno-unknown-warning-option -mthread-model single -isystem "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include" -Ilibc-bottom-half/headers/private -Ilibc-bottom-half/cloudlibc/src/include -Ilibc-bottom-half/cloudlibc/src -Ilibc-top-half/musl/src/include -Ilibc-top-half/musl/src/internal -fPIC -fvisibility=default -MD -MP -o build/wasm32-wasi/libc-bottom-half/crt/crt1-reactor.o -c libc-bottom-half/crt/crt1-reactor.c
clang -O2 -DNDEBUG --target=wasm32-wasi -fno-trapping-math -Wall -Wextra -Werror -Wno-null-pointer-arithmetic -Wno-unused-parameter -Wno-sign-compare -Wno-unused-variable -Wno-unused-function -Wno-ignored-attributes -Wno-missing-braces -Wno-ignored-pragmas -Wno-unused-but-set-variable -Wno-unknown-warning-option -mthread-model single -isystem "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include" -Ilibc-bottom-half/headers/private -Ilibc-bottom-half/cloudlibc/src/include -Ilibc-bottom-half/cloudlibc/src -Ilibc-top-half/musl/src/include -Ilibc-top-half/musl/src/internal -fPIC -fvisibility=default -MD -MP -o build/wasm32-wasi/libc-bottom-half/crt/crt1.o -c libc-bottom-half/crt/crt1.c
#
# Install the startup files (crt1.o etc).
#
mkdir -p "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/lib/wasm32-wasi" && \
        cp build/wasm32-wasi/libc-bottom-half/crt/crt1-command.o build/wasm32-wasi/libc-bottom-half/crt/crt1-reactor.o build/wasm32-wasi/libc-bottom-half/crt/crt1.o "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/lib/wasm32-wasi"
clang -O2 -DNDEBUG --target=wasm32-wasi -fno-trapping-math -Wall -Wextra -Werror -Wno-null-pointer-arithmetic -Wno-unused-parameter -Wno-sign-compare -Wno-unused-variable -Wno-unused-function -Wno-ignored-attributes -Wno-missing-braces -Wno-ignored-pragmas -Wno-unused-but-set-variable -Wno-unknown-warning-option -mthread-model single -isystem "/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include" -Idlmalloc/include -MD -MP -o build/wasm32-wasi/dlmalloc/src/dlmalloc.o -c dlmalloc/src/dlmalloc.c
In file included from dlmalloc/src/dlmalloc.c:63:
In file included from dlmalloc/src/malloc.c:1468:
/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/stdlib.h:85:13: error: unknown type name 'wchar_t'
   85 | int mbtowc (wchar_t *__restrict, const char *__restrict, size_t);
      |             ^
/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/stdlib.h:87:18: error: unknown type name 'wchar_t'
   87 | size_t mbstowcs (wchar_t *__restrict, const char *__restrict, size_t);
      |                  ^
/path/to/wasi/with spaces/sw/libs/wasi-libc/sysroot/include/stdlib.h:88:42: error: unknown type name 'wchar_t'
   88 | size_t wcstombs (char *__restrict, const wchar_t *__restrict, size_t);
      |                                          ^
3 errors generated.
make: *** [build/wasm32-wasi/dlmalloc/src/dlmalloc.o] Error 1

[sbc100]

Can you try running that failing command but adding -E to get the pre-processed output? When I do this and I look at the preprocessed output the first occurrence of wchar_t the I see is:

# 74 "/usr/lib/llvm-14/lib/clang/14.0.6/include/stddef.h" 3
typedef int wchar_t;    

I'm guess you don't see this? Which version of stddef.h is being included in your case? What does the definition of wchar_t look like in that file?