Default to static linking, override with LLVMLITE_SHARED

[tiran]

llvmlite now defaults to statically linking with LLVM. Before llvmlite was dynamically linking with libLLVM when the LLVM supports static and dynamic linking. Most Linux distros have LLVM builds that have static and dynamic linking enabled.

The env var LLVMLITE_SHARED=1 forces dynamic linking with libLLVM.

See: #1027

[tiran]

@geofft @stuartarchibald This PR implements Geoffrey's proposal for standard builds. With my change, llvmlite now defaults to static linking and makes dynamic linking opt-in. We have been using this patch for several months in downstream builds of llvmlite.

I could not figure out how to implement the same feature with CMake build system.

[stuartarchibald]

Thanks for the patch @tiran, this will be discussed at the maintainer's meeting next week.

[stuartarchibald]

I could not figure out how to implement the same feature with CMake build system.

I think it maybe already works (but may need adjusting to be consistent in terms of compiler flags suggested in this PR)?

https://github.com/numba/llvmlite/blob/942e98b4668819280406fcb1065b7bf7e6dd932b/ffi/CMakeLists.txt#L57-L61

[tiran]

I could not figure out how to implement the same feature with CMake build system.

I think it maybe already works (but may need adjusting to be consistent in terms of compiler flags suggested in this PR)?

https://github.com/numba/llvmlite/blob/942e98b4668819280406fcb1065b7bf7e6dd932b/ffi/CMakeLists.txt#L57-L61

No, it does not work in all cases. My system is running Fedora 42. Fedora's LLVM supports both dynamic and static linking. The CMake variable LLVM_AVAILABLE_LIBS contains both all static libraries and the dynamic library LLVM. Because LLVM is listed, the shared object is linked dynamically with LLVM and not statically with the static libs.

/usr/lib64/cmake/llvm/LLVMConfig.cmake:set(LLVM_AVAILABLE_LIBS LLVMDemangle;LLVMSupport;LLVMTableGen;LLVMTableGenBasic;LLVMTableGenCommon;LLVMCore;LLVMFuzzerCLI;LLVMFuzzMutate;LLVMFileCheck;LLVMInterfaceStub;LLVMIRPrinter;LLVMIRReader;LLVMCodeGen;LLVMSelectionDAG;LLVMAsmPrinter;LLVMMIRParser;LLVMGlobalISel;LLVMCodeGenData;LLVMCodeGenTypes;LLVMBinaryFormat;LLVMBitReader;LLVMBitWriter;LLVMBitstreamReader;LLVMDWARFLinker;LLVMDWARFLinkerClassic;LLVMDWARFLinkerParallel;LLVMExtensions;LLVMFrontendDriver;LLVMFrontendHLSL;LLVMFrontendOpenACC;LLVMFrontendOpenMP;LLVMFrontendOffloading;LLVMTransformUtils;LLVMInstrumentation;LLVMAggressiveInstCombine;LLVMInstCombine;LLVMScalarOpts;LLVMipo;LLVMVectorize;LLVMObjCARCOpts;LLVMCoroutines;LLVMCFGuard;LLVMHipStdPar;LLVMLinker;LLVMAnalysis;LLVMLTO;LLVMMC;LLVMMCParser;LLVMMCDisassembler;LLVMMCA;LLVMObjCopy;LLVMObject;LLVMObjectYAML;LLVMOption;LLVMRemarks;LLVMDebuginfod;LLVMDebugInfoDWARF;LLVMDebugInfoGSYM;LLVMDebugInfoLogicalView;LLVMDebugInfoMSF;LLVMDebugInfoCodeView;LLVMDebugInfoPDB;LLVMSymbolize;LLVMDebugInfoBTF;LLVMDWP;LLVMExecutionEngine;LLVMInterpreter;LLVMJITLink;LLVMMCJIT;LLVMOrcJIT;LLVMOrcDebugging;LLVMOrcShared;LLVMOrcTargetProcess;LLVMRuntimeDyld;LLVMPerfJITEvents;LLVMTarget;LLVMAArch64CodeGen;LLVMAArch64AsmParser;LLVMAArch64Disassembler;LLVMAArch64Desc;LLVMAArch64Info;LLVMAArch64Utils;LLVMAMDGPUCodeGen;LLVMAMDGPUAsmParser;LLVMAMDGPUDisassembler;LLVMAMDGPUTargetMCA;LLVMAMDGPUDesc;LLVMAMDGPUInfo;LLVMAMDGPUUtils;LLVMARMCodeGen;LLVMARMAsmParser;LLVMARMDisassembler;LLVMARMDesc;LLVMARMInfo;LLVMARMUtils;LLVMAVRCodeGen;LLVMAVRAsmParser;LLVMAVRDisassembler;LLVMAVRDesc;LLVMAVRInfo;LLVMBPFCodeGen;LLVMBPFAsmParser;LLVMBPFDisassembler;LLVMBPFDesc;LLVMBPFInfo;LLVMHexagonCodeGen;LLVMHexagonAsmParser;LLVMHexagonDisassembler;LLVMHexagonDesc;LLVMHexagonInfo;LLVMLanaiCodeGen;LLVMLanaiAsmParser;LLVMLanaiDisassembler;LLVMLanaiDesc;LLVMLanaiInfo;LLVMLoongArchCodeGen;LLVMLoongArchAsmParser;LLVMLoongArchDisassembler;LLVMLoongArchDesc;LLVMLoongArchInfo;LLVMMipsCodeGen;LLVMMipsAsmParser;LLVMMipsDisassembler;LLVMMipsDesc;LLVMMipsInfo;LLVMMSP430CodeGen;LLVMMSP430Desc;LLVMMSP430Info;LLVMMSP430AsmParser;LLVMMSP430Disassembler;LLVMNVPTXCodeGen;LLVMNVPTXDesc;LLVMNVPTXInfo;LLVMPowerPCCodeGen;LLVMPowerPCAsmParser;LLVMPowerPCDisassembler;LLVMPowerPCDesc;LLVMPowerPCInfo;LLVMRISCVCodeGen;LLVMRISCVAsmParser;LLVMRISCVDisassembler;LLVMRISCVDesc;LLVMRISCVTargetMCA;LLVMRISCVInfo;LLVMSparcCodeGen;LLVMSparcAsmParser;LLVMSparcDisassembler;LLVMSparcDesc;LLVMSparcInfo;LLVMSystemZCodeGen;LLVMSystemZAsmParser;LLVMSystemZDisassembler;LLVMSystemZDesc;LLVMSystemZInfo;LLVMVECodeGen;LLVMVEAsmParser;LLVMVEDisassembler;LLVMVEInfo;LLVMVEDesc;LLVMWebAssemblyCodeGen;LLVMWebAssemblyAsmParser;LLVMWebAssemblyDisassembler;LLVMWebAssemblyDesc;LLVMWebAssemblyInfo;LLVMWebAssemblyUtils;LLVMX86CodeGen;LLVMX86AsmParser;LLVMX86Disassembler;LLVMX86TargetMCA;LLVMX86Desc;LLVMX86Info;LLVMXCoreCodeGen;LLVMXCoreDisassembler;LLVMXCoreDesc;LLVMXCoreInfo;LLVMSandboxIR;LLVMAsmParser;LLVMLineEditor;LLVMProfileData;LLVMCoverage;LLVMPasses;LLVMTargetParser;LLVMTextAPI;LLVMTextAPIBinaryReader;LLVMDlltoolDriver;LLVMLibDriver;LLVMXRay;LLVMTestingAnnotations;LLVMTestingSupport;LLVMWindowsDriver;LLVMWindowsManifest;llvm_gtest;llvm_gtest_main;LTO;LLVMCFIVerify;LLVMDiff;LLVMExegesisX86;LLVMExegesisAArch64;LLVMExegesisPowerPC;LLVMExegesisMips;LLVMExegesis;LLVM;LLVMOptDriver;Remarks)

[stuartarchibald]

I could not figure out how to implement the same feature with CMake build system.

I think it maybe already works (but may need adjusting to be consistent in terms of compiler flags suggested in this PR)? https://github.com/numba/llvmlite/blob/942e98b4668819280406fcb1065b7bf7e6dd932b/ffi/CMakeLists.txt#L57-L61

No, it does not work in all cases. My system is running Fedora 42. Fedora's LLVM supports both dynamic and static linking. The CMake variable LLVM_AVAILABLE_LIBS contains both all static libraries and the dynamic library LLVM. Because LLVM is listed, the shared object is linked dynamically with LLVM and not statically with the static libs.

Many thanks for checking. Given the environment variable in this PR collides with the one supported by CMake builds, I think we'll either need to make them have consistent behaviour, or have separate environment variables. @gmarkall might recall the details of the CMake variant and why it is set to behave this way, I was under the impression that the intended use was to say link statically or link dynamically (if it is not working, it could also just be a bug).

[tiran]

I have been banging my head against the problem. It looks like it's a bug or unsupported scenario in LLVM's CMake. The correct approach should be llvm_map_components_to_libnames(llvm_libs all), but that's still broken in LLVM 15.

I even tried this hack:

    if (NOT ${LLVM_ENABLE_SHARED_LIBS} AND ${LLVM_LINK_LLVM_DYLIB})
        message(STATUS "Removing LLVM from ${llvm_libs}")
        string(REPLACE ";LLVM;" ";" llvm_libs "${llvm_libs}")
    endif()

but it did not work either. The shared library is still linked directly.

c++ -fPIC -fPIC -fno-rtti -g -flto -Wl,--exclude-libs,ALL -shared -Wl,-soname,libllvmlite.so -o libllvmlite.so CMakeFiles/llvmlite.dir/assembly.cpp.o CMakeFiles/llvmlite.dir/bitcode.cpp.o CMakeFiles/llvmlite.dir/core.cpp.o CMakeFiles/llvmlite.dir/initfini.cpp.o CMakeFiles/llvmlite.dir/module.cpp.o CMakeFiles/llvmlite.dir/value.cpp.o CMakeFiles/llvmlite.dir/executionengine.cpp.o CMakeFiles/llvmlite.dir/transforms.cpp.o CMakeFiles/llvmlite.dir/type.cpp.o CMakeFiles/llvmlite.dir/passmanagers.cpp.o CMakeFiles/llvmlite.dir/targets.cpp.o CMakeFiles/llvmlite.dir/dylib.cpp.o CMakeFiles/llvmlite.dir/linker.cpp.o CMakeFiles/llvmlite.dir/object_file.cpp.o CMakeFiles/llvmlite.dir/custom_passes.cpp.o CMakeFiles/llvmlite.dir/orcjit.cpp.o CMakeFiles/llvmlite.dir/memorymanager.cpp.o CMakeFiles/llvmlite.dir/newpassmanagers.cpp.o -Wl,-rpath,/usr/lib64/llvm15/lib [...] /usr/lib64/llvm15/lib/libLLVMSupport.a /usr/lib64/llvm15/lib/libLLVMDemangle.a -lrt -ldl -lm /usr/lib64/libz.so /usr/lib64/libtinfo.so /usr/lib64/llvm15/lib/libLLVM-15.so

[yashssh]

Hi @tiran , #1092 ports llvmlite to LLVM20, can you please try with that and see if the issue persists?

It looks like it's a bug or unsupported scenario in LLVM's CMake. The correct approach should be llvm_map_components_to_libnames(llvm_libs all), but that's still broken in LLVM 15.

[tiran]

Negative

CMake is failing for LLVM 20 on Fedora 43:

$ LLVMLITE_USE_CMAKE=1 python3 -m build -v -n -w
...
-- Configuring done (1.4s)
CMake Error at /usr/lib64/cmake/llvm/LLVMExports.cmake:538 (set_target_properties):
  The link interface of target "LLVMInterpreter" contains:

    FFI::ffi

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

Call Stack (most recent call first):
  /usr/lib64/cmake/llvm/LLVMConfig.cmake:372 (include)
  CMakeLists.txt:20 (find_package)

After hacking LLVMConfig.cmake, it's failing with missing libraries:

[  5%] Linking CXX shared library libllvmlite.so
/usr/bin/ld: cannot find -lLLVMBOLTCore: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTPasses: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTProfile: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTRewrite: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTRuntimeLibs: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTTargetAArch64: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTTargetX86: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTTargetRISCV: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTUtils: No such file or directory

llvm-static-20.1.7-1.fc43.x86_64 does not include the LLVMBOLT archive files, but LLVM_AVAILABLE_LIBS includes the libraries.

Also LLVM_AVAILABLE_LIBS in Fedora's build of LLVM 20 still lists LLVM as a dependency. The build process is going to create a shared library again.

I'll get in contact with our LLVM maintainers.

[yashssh]

/usr/bin/ld: cannot find -lLLVMBOLTCore: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTPasses: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTProfile: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTRewrite: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTRuntimeLibs: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTTargetAArch64: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTTargetX86: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTTargetRISCV: No such file or directory
/usr/bin/ld: cannot find -lLLVMBOLTUtils: No such file or directory

I have a suspicion that these libraries are missing because these projects weren't enabled while building LLVM which you are tying to link against. Maybe you can try with a fresh build of LLVM with below cmake flags set while building LLVM. I am more curious about why llvmlite is looking for these libraries in the first place.

  -DLLVM_ENABLE_PROJECTS="clang;bolt" \
  -DLLVM_TARGETS_TO_BUILD="X86;AArch64;RISCV" \

or better
  -DLLVM_ENABLE_PROJECTS="clang;bolt" \
  -DLLVM_TARGETS_TO_BUILD=all \

More about the flags here https://llvm.org/docs/CMake.html.

[tiran]

The BOLT static archives are not shipped in Fedora builds of LLVM, https://src.fedoraproject.org/rpms/llvm/blob/rawhide/f/llvm.spec#_2079

#region BOLT installation
# We don't ship libLLVMBOLT*.a
rm -f %{buildroot}%{install_libdir}/libLLVMBOLT*.a
#endregion BOLT installation

[stuartarchibald]

I spent quite some time trying to figure out how to use llvm_map_components_to_libnames along with all and came to the conclusion that it probably isn't supported (there are hints in the LLVM CMake scripts that suggest it potentially does not work, but it's not clear). I also don't think using ${LLVM_AVAILABLE_LIBS} will universally work either as it may contain surprising things or reference components that are shipped in some other package (due to a split package build) and also are not actually used by llvmlite. I guess this latter case is what is being described/discovered in the above RE bolt.

As a result of all this, along with other challenges in migrating to LLVM 20, I rewrote the build system so that there is just one system to maintain and deal with (see #1252). It uses solely CMake, defaults to static linkage against LLVM and does so through mapping only the parts of LLVM that llvmlite actually uses into the link libraries. Some of the CI runs are still failing (I need to sort out OSX-arm64 systems, as they don't have the LLVMIntelJITEvents archive for obvious reasons) but I'm hopeful that this will fix the current issues.

The new system also comes with some additional environment variables to help configure the build and some distribution specific tests to help check that the build produced artifacts with expected linkage and other properties.

Once I've got CI working, I'd be interested in seeing how well this works against e.g. Fedora or RHEL packages, along with development or other not-managed-by-Numba-maintainers builds of LLVM.

Hope this helps!

[stuartarchibald]

@tiran #1252 was merged and is in the 0.45.0 release candidates. The new CMake base build system has been tested on a variety of LLVM build configurations and it seems to be doing ok so far. I hope that this will resolve the issues noted above when building against Fedora/RHEL. If you do run a test and come across any issues, please do report them.

As an aside, would it be ok to close this PR? I think the intent (make static linking default) was covered by the much more involved rewrite #1252 which is now on main.

Thanks also for opening this PR, it prompted doing the build system rewrite to fix the issues!

[tiran]

Thanks for the ping and excellent work on the new build system! I'm having some problems with building llvmlite==0.45.0rc2. Let me open a new issue.