llvm
Out of memory trying to link AllClangTests binary using ld.lld on 32-bit Arm Linux
Linaro has a bot armv8-lld-2stage (https://lab.llvm.org/buildbot/#/builders/122) where we built llvm and clang on 32-bit Armv8 Linux. Using lld being the key thing we want to validate.
Since https://github.com/llvm/llvm-project/pull/134196 we have sometimes had this error, and a lot more frequently recently:
FAILED: tools/clang/unittests/AllClangUnitTests
: && /usr/local/bin/c++ -mcpu=neoverse-n1 -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG -fuse-ld=lld -Wl,--color-diagnostics -Wl,--gc-sections tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/Lex/DependencyDirectivesScannerTest.cpp.o <...>tools/clang/unittests/CMakeFiles/AllClangUnitTests.dir/AllClangUnitTests.cpp.o -o tools/clang/unittests/AllClangUnitTests -Wl,-rpath,/home/tcwg-buildbot/worker/clang-armv8-lld-2stage/stage1/lib lib/libLLVMSupport.a lib/libLLVMFrontendHLSL.a lib/libLLVMSupport.a lib/libLLVMAArch64CodeGen.a <...> lib/libLLVMDemangle.a && :
ld.lld: error: failed to open tools/clang/unittests/AllClangUnitTests: Cannot allocate memory
(https://lab.llvm.org/buildbot/#/builders/122/builds/2509 is one build)
I've measured a few builds and ld.lld consistently reaches the 3 point something GB of RAM limit per process.
ld does the link with a peak of ~1.5GB, but using that would be against the point of the bot.
I have tried various options to reduce lld's memory usage, but could not reduce it enough:
- We have threading disabled for the built lld, and I set threads to 1 for the 1st stage lld.
-
--no-mmap-output-file -
-Wl,--build-id=none -Wl,--no-relax(a shot in the dark thinking it might reduce the binary size)
I need to do a survey of all of lld options to find more.
The alternative is to provide a way for builds to opt back into the per-unit-test binary that we had before. I'll post a draft PR that does that for reference purposes.
Rough changes to build single binaries instead: https://github.com/llvm/llvm-project/pull/172685
Appears to work, but only tested on Linux.
To pass options to lld we'll have to modify the bot's config a bit, like this:
$ cmake ../llvm-project/llvm -DLLVM_ENABLE_PROJECTS="llvm;clang" -DCMAKE_BUILD_TYPE=Release \
-DLLVM_ENABLE_ASSERTIONS=ON -G Ninja \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld -Wl,--build-id=none -Wl,--no-relax" \
-DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=lld -Wl,--build-id=none -Wl,--no-relax" \
-DCMAKE_MODULE_LINKER_FLAGS="-fuse-ld=lld -Wl,--build-id=none -Wl,--no-relax"
We cannot use LLVM_USE_LINKER if we're passing lld specific options because cmake's compiler checks will still call ld, which will error.
Passing -fuse-ld= as a compiler flag leads to a lot of "argument unused" when compiling but not linking, so we can solve that by setting all the _LINKER_FLAGS settings.
Our other Armv7/8 32-bit bots do not use lld, so we have not seen this problem there. Which implies that our ld (2.38) is coping well with it.
Disabling ASLR seems to help, but could just be chance:
$ setarch linux32 -R ninja check-clang-unit
Bot moved to silent for now. @antmox and the rest of the Linaro team will deal with it in the new year.