[libc] Fullbuild breaks with clang/musl system due to wint_t

[alfredfo]

Compiling with ninja install-libc

err:

[2/18] Building CXX object projects/libc/src/wchar/CMakeFiles/libc.src.wchar.wctob.dir/wctob.cpp.o
FAILED: projects/libc/src/wchar/CMakeFiles/libc.src.wchar.wctob.dir/wctob.cpp.o 
/usr/lib/llvm/16/bin/clang++-16  --config=/etc/clang/abi-breaking-shit.cfg -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_LIBCPP_ENABLE_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/llvm-project/build-libc-full/projects/libc/src/wchar -I/llvm-project/libc/src/wchar -I/llvm-project/build-libc-full/include -I/llvm-project/llvm/include -I/llvm-project/build-libc-full/projects/libc/include -I/llvm-project/libc -I/llvm-project/build-libc-full/projects/libc -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -g -std=c++17 -fpie -ffreestanding -fno-builtin -fno-exceptions -fno-lax-vector-conversions -fno-unwind-tables -fno-asynchronous-unwind-tables -fno-rtti -Wall -Wextra -Wimplicit-fallthrough -Wwrite-strings -Wextra-semi -Wnewline-eof -Wnonportable-system-include-path -Wstrict-prototypes -Wthread-safety -DLIBC_COPT_PUBLIC_PACKAGING -MD -MT projects/libc/src/wchar/CMakeFiles/libc.src.wchar.wctob.dir/wctob.cpp.o -MF projects/libc/src/wchar/CMakeFiles/libc.src.wchar.wctob.dir/wctob.cpp.o.d -o projects/libc/src/wchar/CMakeFiles/libc.src.wchar.wctob.dir/wctob.cpp.o -c /llvm-project/libc/src/wchar/wctob.cpp
In file included from /llvm-project/libc/src/wchar/wctob.cpp:9:
In file included from /llvm-project/libc/src/wchar/wctob.h:12:
/llvm-project/build-libc-full/projects/libc/include/wchar.h:21:11: error: unknown type name 'wint_t'
int wctob(wint_t) __NOEXCEPT;
          ^
In file included from /llvm-project/libc/src/wchar/wctob.cpp:9:
/llvm-project/libc/src/wchar/wctob.h:16:11: error: unknown type name 'wint_t'
int wctob(wint_t c);
          ^
In file included from /llvm-project/libc/src/wchar/wctob.cpp:11:
/llvm-project/libc/src/__support/wctype_utils.h:28:38: error: unknown type name 'wint_t'
LIBC_INLINE cpp::optional<int> wctob(wint_t c) {
                                     ^
/llvm-project/libc/src/__support/wctype_utils.h:36:27: error: use of undeclared identifier 'wint_t'
LIBC_INLINE cpp::optional<wint_t> btowc(int c) {
                          ^
/llvm-project/libc/src/__support/wctype_utils.h:38:12: error: no viable conversion from returned value of type 'const nullopt_t' to function return type 'int'
    return cpp::nullopt;
           ^~~~~~~~~~~~
/llvm-project/libc/src/__support/wctype_utils.h:39:22: error: unknown type name 'wint_t'
  return static_cast<wint_t>(c);
                     ^
/llvm-project/libc/src/wchar/wctob.cpp:17:1: error: definition 'wctob' cannot also be an alias
LLVM_LIBC_FUNCTION(int, wctob, (wint_t c)) {
^
/llvm-project/libc/src/__support/common.h:38:3: note: expanded from macro 'LLVM_LIBC_FUNCTION'
  LLVM_LIBC_FUNCTION_IMPL(type, name, arglist)
  ^
/llvm-project/libc/src/__support/common.h:30:38: note: expanded from macro 'LLVM_LIBC_FUNCTION_IMPL'
  decltype(__llvm_libc::name) name [[gnu::alias(#name)]];                      \
                                     ^
/llvm-project/libc/src/wchar/wctob.cpp:17:33: error: unknown type name 'wint_t'
LLVM_LIBC_FUNCTION(int, wctob, (wint_t c)) {
                                ^
8 errors generated.

looking at libc/include/llvm-libc-types/wint_t.h we can see that it wants the type wint_t from stddef.h by using the implementation specific __need_wint_t macro. Because this is C++ it will use the libc++ wrapper header that #include_next's stddef.h

libc/include/llvm-libc-types/wint_t.h:

// Since __need_wint_t is defined, we get the definition of wint_t from the
// standalone C header stddef.h. Also, because __need_wint_t is defined,
// including stddef.h will pull only the type wint_t and nothing else.
#define __need_wint_t
#include <stddef.h>
#undef __need_wint_t

We now have two choices, either include the musl stddef.h header, or the internal Clang header.

Clang:

#if defined(__need_wint_t)
/* Always define wint_t when modules are available. */
#if !defined(_WINT_T) || __has_feature(modules)
#if !__has_feature(modules)
#define _WINT_T
#endif
typedef __WINT_TYPE__ wint_t;
#endif
#undef __need_wint_t
#endif /* __need_wint_t */

musl:

#if defined(__NEED_wint_t) && !defined(__DEFINED_wint_t)
typedef unsigned wint_t;
#define __DEFINED_wint_t
#endif

As Clang uses __need_wint_t (underscore) it will work, however it won't with musl as it uses __NEED_wint_t. I am unsure if LLVM libc intends to use the Clang header, or if it depends on glibc internals (it too uses underscore).

Compiling wctob.cpp with the first command + -save-temps, we can see in wctob.ii (preprocessed) that stddef.h comes from the system libc.

$ grep stddef.h wctob.ii 
# 1 "/usr/include/c++/v1/stddef.h" 1 3
# 15 "/usr/include/c++/v1/stddef.h" 3
# 1 "/usr/include/stddef.h" 1 3 4
# 19 "/usr/include/stddef.h" 3 4
# 20 "/usr/include/stddef.h" 2 3 4
# 18 "/usr/include/c++/v1/stddef.h" 2 3
# 1 "/usr/include/c++/v1/stddef.h" 1 3
# 15 "/usr/include/c++/v1/stddef.h" 3
# 1 "/usr/include/c++/v1/stddef.h" 1 3
# 15 "/usr/include/c++/v1/stddef.h" 3
# 1 "/usr/include/c++/v1/stddef.h" 1 3
# 15 "/usr/include/c++/v1/stddef.h" 3

Include path ordering with compilation command + -v

ignoring nonexistent directory "/usr/local/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /llvm-project/build-libc-full/projects/libc/src/wchar
 /llvm-project/libc/src/wchar
 /llvm-project/build-libc-full/include
 /llvm-project/llvm/include
 /llvm-project/build-libc-full/projects/libc/include
 /llvm-project/libc
 /llvm-project/build-libc-full/projects/libc
 /usr/include/c++/v1
 /usr/include
 /usr/lib/llvm/16/bin/../../../../lib/clang/16/include

A very simple and ugly? fix is to just define __NEED_wint_t too. Note that -ffreestanding is used when compiling.

[llvmbot]

@llvm/issue-subscribers-libc

[michaelrj-google]

In general we assume that when we're including the standalone headers (stddef, inttypes, etc.) they are coming from the compiler. This is to avoid this exact sort of compatibility issue with the system's libc. If that isn't what we've been doing then we should figure out why. Is there a way for you to force the clang headers to be higher in the include order, to see if it still works?

[sivachandra]

If -ffreestanding is used, I would expect that the free standing headers are picked up over the system libc headers. If we hold that view, then is it the compiler driver which is breaking the expectation here?

[alfredfo]

@sivachandra: -ffreestanding only means that the build should respect freestanding mode, which musl headers do, not that compiler headers should come before libc headers. In fact, on both Alpine Linux and Gentoo LLVM/musl, the musl headers come first regardless of -ffreestanding.

Found this also: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12477#note_14476, and also discussed in #musl.

I talked with Michael on Discord and the solution here is probably to use musl-specific macros in include/

michaelrj — Today at 12:29 AM
it does look like musl is using their own headers to avoid having to deal with the compiler ones. Given that our libc is somewhat attached to a compiler, I think it's more reasonable for us to expect to use the compiler headers
regardless, I think you could adjust the type headers in include/ to have musl handling

catcream — Today at 12:31 AM
Yeah, my idea is that we should use some CMake magic to force Clang headers to be used first unconditionally, not rely on -ffreestanding

catcream — Today at 12:31 AM
aha

michaelrj — Today at 12:31 AM
specifically libc/include/llvm-libc-types/wint_t.h you could probably have it define both __need_wint_t and __NEED_wint_t if you add a comment

catcream — Today at 12:31 AM
yes!
though I would assume this would be an issue going forward
since you need to specifically use musl internals everytime it's not compatible with Clang

michaelrj — Today at 12:33 AM
I guess it depends on how often it's incompatible with clang. Adding one or two extra #defines isn't too bad, but if it's a common problem then we'd probably want to handle it like we do gcc as a different target with its own buildbot

catcream — Today at 12:34 AM
that makes sense yeah, I know there was some other error popping up after fixing this
maybe I should actually get it to build, and then decide to do one or the other?
so I know how much this pops up I mean

michaelrj — Today at 12:36 AM
yeah, it's probably going to be wint_t and wchar if I had to guess

[sivachandra]

@alfredfo - I am sorry I will not be able to participate in discord discussions for a few days more.

Thanks for sharing the references about freestanding implementations. The LLVM libc full build is intended to be a mode in which LLVM's libc is the only libc in use. This leads to few interesting points:

1. musl or glibc are likely the default libc on a system, but when using LLVM's libc in the full build mode, LLVM's libc is the system libc.
2. It is incorrect to mix or use the default system libc's headers when LLVM's libc is the system libc.
3. Compiler drivers order the system include paths based on the system libc that is in use. So, when using LLVM's libc as the system libc, they should do what LLVM's libc wants.

In the long run, compiler drivers have to be taught about LLVM's libc. May be there already is a driver option to switch the include path order. If not, then like how musl provides musl-clang and musl-gcc wrapper scripts, I think we have to use similar scripts for LLVM's libc on musl like systems.

[androm3da]

In the long run, compiler drivers have to be taught about LLVM's libc

On a related note -- what's the triple component for LLVM libc we should be using? I'm doing the cross build using hexagon-unknown-linux-musl but it seems like I should use something like hexagon-unknown-linux-llvm instead?

It makes sense to agree on this so we know how to change the relevant driver(s).

[sivachandra]

<arch>-unknonw-linux-llvm sounds reasonable to me.

[nickdesaulniers]

When building llvmlibc, we should NOT be including the system headers of the host EVER.

A very simple and ugly? fix is to just define __NEED_wint_t too.

In libc/include/llvm-libc-types/wint_t.h?

Note that -ffreestanding is used when compiling.

I wonder if we need to add -nostdinc to the build flags to get this behavior? (We need to avoid having the system libc headers in the inclusion search path, but not the compiler provided ones like stddef.h)

[nickdesaulniers]

adding -nostdinc -I$(clang -print-resource-dir)/include to the invocation of clang allowed me to avoid this issue, but doesn't work across the remainder of the build.

basically we want to:

1. tell clang to forget about all the implicit default include paths (-nostdinc) (this forgets about the compiler provided headers)
2. re-add the compiler provided headers via -I$(clang -print-resource-dir)/include

[nickdesaulniers]

The issue stems from the fact that the llvmlibc build isn't as hermetic as I'd like. The build relies on the kernel headers being installed in:

1. /usr/include/linux/
2. /usr/include/asm/
3. /usr/include/asm-generic/

but then it can't differentiate when musl puts its stddef.h in

• /usr/include/stddef.h

It's problematic to rely on the host's kernel headers anyways, since it will lead to failure to build (at best) or super subtle bugs at runtime (at worst) if you're trying to cross compile llvmlibc anyways.

To me, for any libc, I'd build the kernel headers first, put them in a non-standard location, then tell the libc's build system to:

1. not use the standard implicit place to look for headers (-nostdinc)
2. use this explicitly non standards place to look for kernel headers -system or --sysroot or -idirafter.

[nickdesaulniers]

I think this demonstrates the lack of hermiticity. On my x86_64-linux-gnu host, if I build with ninja -d keepdepfile libc in order for ninja not to delete the .d dependency files, then <build dir>/projects/libc/src/wchar/CMakeFiles/libc.src.wchar.wctob.dir/wctob.cpp.o.d contains:

/android0/aosp/prebuilts/clang/host/linux-x86/clang-r498229b/lib/clang/17/include/stddef.h \
/android0/aosp/prebuilts/clang/host/linux-x86/clang-r498229b/lib/clang/17/include/__stddef_max_align_t.h \     
/android0/aosp/prebuilts/clang/host/linux-x86/clang-r498229b/lib/clang/17/include/float.h \
/android0/aosp/prebuilts/clang/host/linux-x86/clang-r498229b/lib/clang/17/include/stdarg.h \

those paths are from the compiler I used to bootstrap the build! (via -DCMAKE_C_COMPILER). The rest look good, but those 4 do not!

[alfredfo]

The issue stems from the fact that the llvmlibc build isn't as hermetic as I'd like. The build relies on the kernel headers being installed in:

1. /usr/include/linux/
2. /usr/include/asm/
3. /usr/include/asm-generic/

but then it can't differentiate when musl puts its stddef.h in

• /usr/include/stddef.h

It's problematic to rely on the host's kernel headers anyways, since it will lead to failure to build (at best) or super subtle bugs at runtime (at worst) if you're trying to cross compile llvmlibc anyways.

To me, for any libc, I'd build the kernel headers first, put them in a non-standard location, then tell the libc's build system to:

1. not use the standard implicit place to look for headers (-nostdinc)
2. use this explicitly non standards place to look for kernel headers -system or --sysroot or -idirafter.

Yes, me and Siva discussed this in #libc and one solution is to do -nostdinc and provide kernel/OS headers in a separate directory (not using /usr/include) as you said. This is also an issue when you make LLVM-libc headers with the same name as Clang provided ones btw. I added limits.h a while back, and because it uses include_next for the compiler's limits.h it will include /usr/include/limits.h into the LLVM-libc build if you don't use -nostdinc.