clang++ shipped with the wasi-sdk errors with LLVM ERROR: out of memory - clang++ shipped with emscripten is ok

[yowl]

Using the clang++ shipped with wask-sdk, which reports as 16.0.0:

E:\tmp\consolelocal>%WASI_SDK_PATH%\bin\clang++.exe --version
clang version 16.0.0
Target: wasm32-unknown-wasi
Thread model: posix
InstalledDir: e:/github/wasi-sdk/bin

it fails on windows with some input (a 25MB .bc file) with

E:\tmp\consolelocal>%WASI_SDK_PATH%\bin\clang++.exe  -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=E:\GitHub\wasi-sdk\share\wasi-sysroot -c -g3 obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.bc -o obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.o -pthread --sysroot=%WASI_SDK_PATH%/share/wasi-sysroot -target wasm32-wasi-threads
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
LLVM ERROR: out of memory
Allocation failed
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: e:\\github\\wasi-sdk\\bin\\clang++.exe -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=E:\\GitHub\\wasi-sdk\\share\\wasi-sysroot -c -g3 obj\\Debug\\net8.0\\wasi-wasm\\native\\consoleclrjit.bc -o obj\\Debug\\net8.0\\wasi-wasm\\native\\consoleclrjit.o -pthread --sysroot=e:\\github\\wasi-sdk/share/wasi-sysroot -target wasm32-wasi-threads
1.      Code generation
2.      Running pass 'Function Pass Manager' on module 'obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.bc'.
3.      Running pass 'WebAssembly Instruction Selection' on function '@S_P_CoreLib_System_Globalization_UmAlQuraCalendar__InitDateMapping'
Exception Code: 0xC000001D
#0 0x0088fb73
#1 0x75b9e332
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 16.0.0
Target: wasm32-unknown-wasi-threads
Thread model: posix
InstalledDir: e:/github/wasi-sdk/bin
clang++: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.

With emscripten's clang++ (also 16.0.0, but maybe a different commit):

E:\tmp\consolelocal>%EMSDK%\upstream\bin\clang++.exe --version
clang version 16.0.0 (https://github.com/llvm/llvm-project 8b587113b746f31b63fd6473083df78cef30a72e)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: E:\GitHub\emsdk\upstream\bin

This passes:

E:\tmp\consolelocal>%EMSDK%\upstream\bin\clang++.exe  -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=E:\GitHub\wasi-sdk\share\wasi-sysroot -c -g3 obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.bc -o obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.o -pthread --sysroot=%WASI_SDK_PATH%/share/wasi-sysroot -target wasm32-wasi-threads
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
1 warning generated.

The LLVM for the function that is failing is attached, but it might not be that relevant, maybe its just run out of memory in general, this function appears 74% down the input. Also attached the complete .bc input.

S_P_CoreLib_System_Globalization_UmAlQuraCalendar__InitDateMapping.txt consoleclrjit.zip

[sbc100]

There are several reasons such a difference between the two compiler binaries might exist. Firstly, the emscripten binaries are build against LLVM tip-of-tree so they are built from very different set of sources. Secondly they are built with LTO which would effect all kind of things, including memory usage.

How much memory does the machine your running on have? Does the issue still occur on machines with a lot more memory? (i.e. is this just a run of mill OOM, or is it some kind of bug?)

[yowl]

64GB, when I run it the memory graph in Task Manager shows just a small spike( not a great indicator, but something):

[yowl]

I don't have anything with more than 64GB, unfortunately.

[sbc100]

How much memory does the emscripten version clang use?

I guess you could try building wasi-sdk with clang tip-of-tree to see if there is perhaps a bugfix?

[yowl]

With emscripten it barely moves

I can try clang tip-of-tree with wasi-sdk, or when do you think v21 will be out?

[sbc100]

wasi-sdk doesn't use tip-of-tree llvm, so the next major llvm update would be when llvm 17 is released (which could be a while).

[yowl]

If llvm 16 accepted a bug fix (assuming that's what it is), would wasi-sdk take it?

[sunfishcode]

Yes, if a fix is backported to the release/16.x branch, wasi-sdk can update to it.

[yowl]

Thanks, it looks like upstream is fine. There is clearly a difference between clang++ from the one from Wasi-SDK and https://github.com/llvm/llvm-project/releases/download/llvmorg-16.0.0/LLVM-16.0.0-win64.exe. Here is the output from Wasi-SDK's clang++ and LLVM's clang++ . Are there any options I could pass to LLVM's version to make it behave like the one from Wasi-SDK?

E:\tmp\consolelocal>"c:\Program Files\LLVM\bin\clang++.exe" "obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.bc" -o "obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.o" -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot="e:\github\wasi-sdk/share/wasi-sysroot" -pthread -target wasm32-wasi-threads -c  -g3
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
1 warning generated.

E:\tmp\consolelocal>%WASI_SDK_PATH%\bin\clang++.exe "obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.bc" -o "obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.o" -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot="e:\github\wasi-sdk/share/wasi-sysroot" -pthread -target wasm32-wasi-threads -c  -g3
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
LLVM ERROR: out of memory
Allocation failed
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: e:\\github\\wasi-sdk\\bin\\clang++.exe obj\\Debug\\net8.0\\wasi-wasm\\native\\consoleclrjit.bc -o obj\\Debug\\net8.0\\wasi-wasm\\native\\consoleclrjit.o -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=e:\\github\\wasi-sdk/share/wasi-sysroot -pthread -target wasm32-wasi-threads -c -g3
1.      Code generation
2.      Running pass 'Function Pass Manager' on module 'obj\Debug\net8.0\wasi-wasm\native\consoleclrjit.bc'.
3.      Running pass 'WebAssembly Instruction Selection' on function '@S_P_CoreLib_System_Reflection_Runtime_MethodInfos_RuntimeConstructedGenericMethodInfo__get_ReflectedType'
Exception Code: 0xC000001D
0x0084FB73 <unknown module>
0x7649E332 <unknown module>
0x008323A1 <unknown module>
0x008344C0 <unknown module>
0x010F2CA9 <unknown module>
0x010F1ADD <unknown module>
0x010DB8F9 <unknown module>
0x7732BAFE <unknown module>
0x010DB8AA <unknown module>
0x010EF651 <unknown module>
0x009DA3AB <unknown module>
0x00695AD4 <unknown module>
0x00698B5B <unknown module>
0x00E18697 <unknown module>
0x01043B12 <unknown module>
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 16.0.0
Target: wasm32-unknown-wasi-threads
Thread model: posix
InstalledDir: e:/github/wasi-sdk/bin
clang++: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.

[abrown]

@sunfishcode and I were discussing this: before we decide what to do here we need to understand better what is the difference in behavior between this LLVMv16 Clang and the wasi-sdk Clang. One suggestion would be to compile with -S with both versions and see if there is any difference in that output?

[yowl]

I could try -S on a simple C program, I can't do it on this input as the SDK clang++ fails with LLVM ERROR: out of memory

[yowl]

Only thing I've noticed that is different is for WASI SDK the default target is different:

clang -cc1 version 16.0.0 based upon LLVM 16.0.0 default target wasm32-wasi

And the LLVM download:

clang -cc1 version 16.0.0 based upon LLVM 16.0.0 default target x86_64-pc-windows-msvc

Probably not interesting though?

[yowl]

Small update, clang++ from the wasi-sdk for Linux is also fine:

scott@Matumbo:~/clang-repro$ ../wasi-sdk-20.0/bin/clang++ consoleclrjit.bc -o consoleclrjit.o -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot="e:\github\wasi-sdk/share/wasi-sysroot" -pthread -target wasm32-wasi-threads -c  -g3
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
1 warning generated.

So problem seems isolated to the Windows build.

[yowl]

Tried with sdk 21 and problem persists:

C:\tmp>%WASI_SDK_PATH%\bin\clang++.exe  -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=c:\GitHub\wasi-sdk21\share\wasi-sysroot -c -g3 consoleclrjit.bc -o consoleclrjit.o -pthread --sysroot=%WASI_SDK_PATH%/share/wasi-sysroot -target wasm32-wasi-threads
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
LLVM ERROR: out of memory
Allocation failed
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: c:\\github\\wasi-sdk21\\bin\\clang++.exe -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=c:\\GitHub\\wasi-sdk21\\share\\wasi-sysroot -c -g3 consoleclrjit.bc -o consoleclrjit.o -pthread --sysroot=c:\\github\\wasi-sdk21/share/wasi-sysroot -target wasm32-wasi-threads
1.      Code generation
2.      Running pass 'Function Pass Manager' on module 'consoleclrjit.bc'.
3.      Running pass 'WebAssembly Explicit Locals' on function '@S_P_TypeLoader_Internal_TypeSystem_MethodSignature__Equals_0'
Exception Code: 0xC000001D
 #0 0x002f4c15
 #1 0x76dce332
 #2 0x002d6d34
 #3 0x002d8de4
 #4 0x00b15963
 #5 0x00b15cd8
 #6 0x00b15c42
 #7 0x00b15bd4
 #8 0x004496fa
 #9 0x028a7400
#10 0x027f328c
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 17.0.6
Target: wasm32-unknown-wasi-threads
Thread model: posix
InstalledDir: c:/github/wasi-sdk21/bin
clang++: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.

[yowl]

More info, the x86 version of clang++ fails as above, the x64 version passes. Is there a reason the SDK uses the 32 bit version, do LLVM even support that properly ?

[yowl]

Wasi-SDK includes the 32 bit version.

[abrown]

I checked and this is not the case:

$ file wasi-sdk-21.0/bin/clang-17
wasi-sdk-21.0/bin/clang-17: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=2b3af36d4a17f9cb2a3c33a3d530eece25498ba6, stripped

At least not on Linux, but now I see that your examples are all Windows-based. Can you confirm via some shell output that in fact those binaries are all 32-bit on Windows? Seems a bit weird but maybe there's some Makefile setting that needs to be tweaked (?).

[yowl]

Sure:

C:\github\runtimelab>dumpbin /headers %WASI_SDK_PATH%\bin\clang.exe
Microsoft (R) COFF/PE Dumper Version 14.38.33135.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file c:\github\wasi-sdk21\bin\clang.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)
               9 number of sections
        657CFC0A time date stamp Fri Dec 15 20:23:22 2023
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             102 characteristics
                   Executable
                   32 bit word machine

[abrown]

Ok, I think this makes things more clear. What might be happening here is that the script that downloads the release artifacts is first extracting the x64 version and then overwriting this with the x86 version :

https://github.com/WebAssembly/wasi-sdk/blob/317548590b402d622194416094fe0e392b725fe2/ci/download-workflow-artifacts.sh#L61

Maybe a fix for this is to do something similar to the exclusion added to skip one of the Ubuntu versions:

https://github.com/WebAssembly/wasi-sdk/blob/317548590b402d622194416094fe0e392b725fe2/ci/download-workflow-artifacts.sh#L43-L45

One way to check whether all of this is in fact the case would be to retrieve the x64 binary from this workflow: https://github.com/WebAssembly/wasi-sdk/actions/runs/7228245836. That workflow is the one that was used for creating the wasi-sdk-21 release in which I am guessing we used the x86 version instead. Can you see if the clang in the dist-windows-latest-x64 zip file fixes the issue you were seeing?

[yowl]

Yes, I confirm that clang fixes the issue

C:\tmp>"C:\Users\scott\Downloads\wasi-sdk-21.0+m-mingw.tarx\wasi-sdk-21.0+m\bin\clang.exe" -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -disable-lsr --sysroot=E:\GitHub\wasi-sdk\share\wasi-sysroot -c -g3 c:\tmp\consoleclrjit.bc -o c:\tmp\consoleclrjit.o -pthread --sysroot=%WASI_SDK_PATH%/share/wasi-sysroot -target wasm32-wasi-threads
warning: overriding the module target triple with wasm32-unknown-wasi-threads [-Woverride-module]
1 warning generated.

C:\tmp>

[dicej]

What's left to do here? Do we need to make a 22.1 release that includes the x64 build of clang etc. instead of the x86 build?

[abrown]

This fell off my radar: the issue felt resolved in my mind although in fact no one ever modified download-workflow-artifacts.sh. @dicej, are you interested in resolving this? I think the first thing to do is to modify that script so that future releases use the right binary. After that, yeah, we could draft a new release or probably even modify the release artifacts for wasi-sdk-22 (?).

[dicej]

Yeah, I'll put it on my to-do list and open a PR today or tomorrow.