packages
packages copied to clipboard
getsolus
Freeing Clang from GNU tooling (T6774)
LLVM provides alternatives for the majority of GNU tools, most of which we package but we don't actually use when we specify clang : yes in a package. Additionally, we build clang using GNU tooling when it could be built with clang's own tools. Let's list them to see what we could potentially implement.
- [x] AR, NM and RANLIB
LLVM provides
llvm-ar,llvm-nmandllvm-ranlibas alternatives to GNU'sar,nmandranlib. They reportably perform better but with some limitations, e.g.llvm-nmdoes not accept the same full set of arguments asnm. They can be set at build time by exporting theARNMandRANLIBglobal variables.
Even though this is implemented LLVM itself is still building with GCC tools here, could be a cmake thing, can set CMAKE_RANLIB and so on to circumvent.
-
[x] Linker We already package clang's linker
lld, it can be used by specifying-fuse-ld=lldin LDFLAGS. Clang can be built with it's own linker withLLVM_ENABLE_LLDorLLVM_USE_LINKER=lldat build time. Importantly it doesn't understand our default LDFLAG--copy-dt-needed-entries, removing this as default will likely bloat builddeps for rebuilds. -
[x] C library Whilst it is possible to use
muslinstead ofglibcwith clang we want to take advantage of the AVX2 architecture we already have in place. No point in changing this.
C++ library
LLVM provides libc++ which we already have packaged as a drop-in replacement to GNU's libstdc++. It can be used at build time by specifying -stdlib=libc++. It can be made clang's default C++ library by setting CLANG_DEFAULT_CXX_STDLIB=libc++ when building Clang.
You can build llvm with libcxx by passing LLVM_ENABLE_LIBCXX
LLVM's build system looks for an installed libcxx installation rather than using the in-tree compiled source, it's possible to overcome this with some trickery but it may be worth moving libcxx outside of the monolithic llvm build. This also applies to libcxxabi and libunwind.
Compiler Runtime (and atomics)
LLVM provides compiler-rt as a drop in replacement to libgcc_s. It can be used by specifying -rtlib=compiler-rt at build time, alternatively you can set clang's default compiler runtime by setting CLANG_DEFAULT_RTLIB=compiler-rt when building LLVM.
C++ ABI
LLVM provides libc++abi as a drop-in replacement to GNU's libsupc++ which is contained within libstdc++.
You can bootstrap libc++ with libc++abi by setting LIBCXX_CXX_ABI=libcxxabi.
You can build llvm with libcxxabi by passing LLVM_ENABLE_LIBCXXABI.
Unwind Library
There exists three unwind libraries. The one included within libgcc_s, libunwind from non-gnu and LLVM's own libunwind which we currently do not have packaged, meant for use with libcxxabi. LLVM's libunwind will conflict with non-gnu's libunwind library.
Building Clang with PGO
We want clang to compile faster, especially as we already have GCC PGO. Their provided pgo solution requires some serious cmake gymnastics to build a monolithic LLVM stack but it's fairly simple to build LLVM by: building a stage1 compiler as normal, building a stage2 instrumented compiler with LLVM_BUILD_INSTRUMENTED, build the profile, build the stage2 profiled compiler with the stage1 compiler and point it to the profile with LLVM_PROFDATA_FILE. This allows us to keep a clean environment and choose what we want to run to build a profile. Either running check-clang or compiling clang
https://llvm.org/docs/AdvancedBuilds.html#multi-stage-pgo
Whilst early technology it may be worth looking at BOLT which provides an additional 12% compile time improvement over a PGO'd and LTO'd build.
Why To get the most performance out of clang and make it a real replacement to gcc.
Resources http://bcain-llvm.readthedocs.io/projects/clang/en/latest/Toolchain/#assembling-a-complete-toolchain https://blogs.gentoo.org/gsoc2016-native-clang/ https://archive.fosdem.org/2018/schedule/event/crosscompile/attachments/slides/2107/export/events/attachments/crosscompile/slides/2107/How_to_cross_compile_with_LLVM_based_tools.pdf
There is still some missing information here but opening this up in-case it becomes useful - mostly to sunnyflunk. Though, I'm sure he already knows this stuff. This can also be used for tracking.
AR, NM and RANLIB Tick,
ypkgwill already do this (though it's mainly targeted for clang 7)
Importantly it doesn't understand our default LDFLAG --copy-dt-needed-entries
ypkgalready handles this forthin-ltoandclangbuilds
CLANG_DEFAULT_CXX_STDLIB=libc++ Did build
libflacwith a test build and linked againstlibc++libs
CLANG_DEFAULT_RTLIB=compiler-rt Think I ended up removing this due to build trouble, but
libflacdidn't dep onlibgcc_sso unsure if it works by default.
Unwind Library Currently built statically to avoid conflicts. I was hoping it wouldn't need to be shipped, but it may require that.
Building Clang with PGO I still believe without adding perf training data to the build (stuff for it to compile in a format I didn't really get), it provides no benefit to perform the PGO build. Will likely be better off building a PGO-generate version and then compiling a couple of programs (C and C++) and use that as the data.
Other notes:
I have considered splitting the build into its separate parts (so it can be made more sane), though this will break the compiler every major update of llvm/clang, and parts will be need to be built twice, but only some parts needs avx2 builds. At the very least, lldb will be kicked out as it adds no value being in the build and can be used externally
The other issue is dealing with LD_AS_NEEDED with the alternative linkers, as it won't understand it.
These patches to lld should bring the build to match the behaviour of the current binutils linkers and Solus setup. Haven't tested they build yet.
https://hastebin.com/ituvahutey
Re-opening since some things were rolled back
Ideally we need to validate that AR, NM, RANLIB are being understood by all build types (configure, cmake, meson etc).
My thought on C++ is that the C++1y option may be problematic for the extra libraries to work. So trying it again without that might be beneficial. If they aren't used in the build, then perhaps they should be separated. Such changes would require a lot of testing.
I am considering making llvm static if it is able to improve performance sufficiently. Clang and Mesa performance especially (with LTO).
As for BOLT, it will take quite some plumbing to incorporate into builds well. It also seems to be best in large statically linked binaries (rather than libraries), so its use may be limited given most parts of programs are often shared libraries. More interested in it for GCC currently as that will actually make a difference and worth the effort if it were to speed up builds 10%.
It also should be noted LLD has still many bugs and even with version 8 many libraries will be completly broken, even more libraries will seem ok but will segfault in specific conditions. Running testsuite is reliable way to detect it but there are many packages without check step.
