bcc
bcc copied to clipboard
iovisor
combine static libraries into `bcc`
When linking BCC statically to a program, some symbols are missing (output truncated, longer output at the bottom of this message):
undefined symbol: ebpf::BPF::get_syscall_fnname(...
undefined symbol: ebpf::BPF::init(...
undefined symbol: ebpf::BPF::~BPF(...
undefined symbol: ebpf::BPF::attach_kprobe(...
undefined symbol: ebpf::BPFTable::BPFTable(...
undefined symbol: ebpf::BPFModule::BPFModule(...
undefined symbol: ebpf::BPF::detach_kprobe(...
undefined symbol: ebpf::ProgFuncInfo::get_func(...
undefined symbol: ebpf::ProgFuncInfo::for_each_func(...
undefined symbol: ebpf::ClangLoader::~ClangLoader(...
undefined symbol: ebpf::ProgFuncInfo::func_name[abi:cxx11](...
undefined symbol: ebpf::ProgFuncInfo::get_func(...
undefined symbol: ebpf::ClangLoader::ClangLoader(...
undefined symbol: ebpf::ProgFuncInfo::add_func(...
undefined symbol: ebpf::ClangLoader::parse(...
Running nm --demangle --defined-only -g on libbcc.a confirms those
symbols are not defined by that library.
After looking at the bcc repository and build directory, I've found a
file named libapi-static.a and libclang_frontend.a, which do contain
the necessary symbols. These libraries don't get installed by make install, though.
Double checking src/cc/api/CMakeLists.txt and
src/cc/frontends/clang/CMakeLists.txt confirms that the corresponding
targets api-static and clang_frontend are never scheduled for
installation.
After manually installing these files and linking against them, the linker errors go away.
This patch originally made the necessary changes to CMakeLists.txt so
that the targets get installed under the names libbcc_api.a and
libbcc_clang_frontend.a. Later it was decided that the libraries
should be merged together into libbcc, which the final form of the
patch does.
FWIW, these are the build flags passed to cmake when building bcc
(shared libraries are disabled):
cmake
-DENABLE_CLANG_JIT=ON
-DENABLE_CPP_API=ON
-DENABLE_EXAMPLES=OFF
-DENABLE_LLVM_NATIVECODEGEN=ON
-DENABLE_LLVM_SHARED=OFF
-DENABLE_MAN=ON
-DENABLE_RTTI=OFF
-DENABLE_TESTS=OFF
-DENABLE_USDT=ON
-DPYTHON_ONLY=OFF
-DRUN_LUA_TESTS=OFF
-DBUILD_SHARED_LIBS=OFF
-DCMAKE_BUILD_TYPE=Release
-DCMAKE_CXX_STANDARD:STRING="20"
-DCMAKE_INSTALL_PREFIX:PATH=/usr/local
..
Linker errors:
+ /usr/local/bin/clang++ -pthread -fuse-ld=mold -static-libgcc -static /src/out/main.o -o /src/out/main /usr/local/lib/libbcc.a /usr/local/lib/libbcc_bpf.a /usr/local/lib/libbcc-loader-static.a /usr/local/lib/libclangFrontend.a /usr/local/lib/libclangSerialization.a /usr/local/lib/libclangDriver.a /usr/local/lib/libclangASTMatchers.a /usr/local/lib/libclangParse.a /usr/local/lib/libclangSema.a /usr/local/lib/libclangCodeGen.a /usr/local/lib/libclangAnalysis.a /usr/local/lib/libclangRewrite.a /usr/local/lib/libclangEdit.a /usr/local/lib/libclangIndex.a /usr/local/lib/libclangAST.a /usr/local/lib/libclangLex.a /usr/local/lib/libclangBasic.a /usr/local/lib/libclang.a /usr/local/lib/libLLVMBPFDisassembler.a /usr/local/lib/libLLVMBPFAsmParser.a /usr/local/lib/libLLVMBPFCodeGen.a /usr/local/lib/libLLVMBPFDesc.a /usr/local/lib/libLLVMBPFInfo.a /usr/local/lib/libLLVMCoverage.a /usr/local/lib/libLLVMX86Disassembler.a /usr/local/lib/libLLVMX86AsmParser.a /usr/local/lib/libLLVMX86CodeGen.a /usr/local/lib/libLLVMX86Desc.a /usr/local/lib/libLLVMX86Info.a /usr/local/lib/libLLVMOrcJIT.a /usr/local/lib/libLLVMMCJIT.a /usr/local/lib/libLLVMExecutionEngine.a /usr/local/lib/libLLVMRuntimeDyld.a /usr/local/lib/libLLVMOrcTargetProcess.a /usr/local/lib/libLLVMOption.a /usr/local/lib/libLLVMMCDisassembler.a /usr/local/lib/libLLVMLTO.a /usr/local/lib/libLLVMPasses.a /usr/local/lib/libLLVMCFGuard.a /usr/local/lib/libLLVMCoroutines.a /usr/local/lib/libLLVMObjCARCOpts.a /usr/local/lib/libLLVMipo.a /usr/local/lib/libLLVMVectorize.a /usr/local/lib/libLLVMLinker.a /usr/local/lib/libLLVMInstrumentation.a /usr/local/lib/libLLVMFrontendOpenMP.a /usr/local/lib/libLLVMGlobalISel.a /usr/local/lib/libLLVMMIRParser.a /usr/local/lib/libLLVMAsmPrinter.a /usr/local/lib/libLLVMSelectionDAG.a /usr/local/lib/libLLVMCodeGen.a /usr/local/lib/libLLVMIRReader.a /usr/local/lib/libLLVMAsmParser.a /usr/local/lib/libLLVMTarget.a /usr/local/lib/libLLVMScalarOpts.a /usr/local/lib/libLLVMInstCombine.a /usr/local/lib/libLLVMAggressiveInstCombine.a /usr/local/lib/libLLVMTransformUtils.a /usr/local/lib/libLLVMBitWriter.a /usr/local/lib/libLLVMAnalysis.a /usr/local/lib/libLLVMProfileData.a /usr/local/lib/libLLVMDebugInfoDWARF.a /usr/local/lib/libLLVMObject.a /usr/local/lib/libLLVMTextAPI.a /usr/local/lib/libLLVMMCParser.a /usr/local/lib/libLLVMMC.a /usr/local/lib/libLLVMDebugInfoCodeView.a /usr/local/lib/libLLVMBitReader.a /usr/local/lib/libLLVMCore.a /usr/local/lib/libLLVMRemarks.a /usr/local/lib/libLLVMBitstreamReader.a /usr/local/lib/libLLVMBinaryFormat.a /usr/local/lib/libLLVMTableGen.a /usr/local/lib/libLLVMSupport.a /usr/local/lib/libLLVMDemangle.a /usr/local/lib/libxml2.a /usr/local/lib/libncurses.a /usr/local/lib/libtinfo.a /usr/local/lib64/libbpf.a /usr/local/lib/libelf.a /usr/local/lib/libdw.a /usr/local/lib/libasm.a /usr/local/lib/libzstd.a /usr/local/lib/libz.a -ldl
...
mold: error: undefined symbol: ebpf::BPF::get_syscall_fnname(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
>>> referenced by kprobe.cpp
>>> /src/out/libkprobe.a(kprobe.cpp.o):(kprobe::kprobe(program&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))
mold: error: undefined symbol: ebpf::BPF::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<ebpf::USDT, std::allocator<ebpf::USDT> > const&)
>>> referenced by program.cpp
>>> /src/out/libprogram.a(program.cpp.o):(program::program(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >))
mold: error: undefined symbol: ebpf::BPF::~BPF()
>>> referenced by program.cpp
>>> /src/out/libprogram.a(program.cpp.o):(program::~program())
mold: error: undefined symbol: ebpf::BPF::attach_kprobe(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bpf_probe_attach_type, int)
>>> referenced by kprobe.cpp
>>> /src/out/libkprobe.a(kprobe.cpp.o):(kprobe::kprobe(program&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))
mold: error: undefined symbol: ebpf::BPFTable::BPFTable(ebpf::TableDesc const&)
>>> referenced by program.cpp
>>> /src/out/libprogram.a(program.cpp.o):(ebpf::BPF::get_table(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))>>> referenced by program.cpp
>>> /src/out/libprogram.a(program.cpp.o):(ebpf::BPF::get_table(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))
mold: error: undefined symbol: ebpf::BPFModule::BPFModule(unsigned int, ebpf::TableStorage*, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, char const*)
>>> referenced by program.cpp
>>> /src/out/libprogram.a(program.cpp.o):(ebpf::BPF::BPF(unsigned int, ebpf::TableStorage*, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool))
mold: error: undefined symbol: ebpf::BPF::detach_kprobe(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bpf_probe_attach_type)
>>> referenced by kprobe.cpp
>>> /src/out/libkprobe.a(kprobe.cpp.o):(kprobe::detach())
mold: error: undefined symbol: ebpf::ProgFuncInfo::get_func(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::function_start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const)>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::function_source(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const)>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::function_source_rewritten(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const)>>> referenced 2 more times
mold: error: undefined symbol: ebpf::ProgFuncInfo::for_each_func(std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ebpf::FuncInfo&)>)
>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::~BPFModule())>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::load_maps(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::tuple<unsigned char*, unsigned long, unsigned int>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::tuple<unsigned char*, unsigned long, unsigned int> > > >&))>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::finalize())>>> referenced 3 more times
mold: error: undefined symbol: ebpf::ClangLoader::~ClangLoader()
>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::load_cfile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, char const**, int))>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::load_cfile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, char const**, int))>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::load_includes(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))>>> referenced 6 more times
mold: error: undefined symbol: ebpf::ProgFuncInfo::func_name[abi:cxx11](unsigned long)
>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::function_name(unsigned long) const)
mold: error: undefined symbol: ebpf::ProgFuncInfo::get_func(unsigned long)
>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::function_start(unsigned long) const)>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::function_size(unsigned long) const)
mold: error: undefined symbol: ebpf::ClangLoader::ClangLoader(llvm::LLVMContext*, unsigned int)
>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::load_cfile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, char const**, int))>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::load_includes(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::load_c(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const**, int))>>> referenced 1 more times
mold: error: undefined symbol: ebpf::ProgFuncInfo::add_func(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(std::_Function_handler<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ebpf::FuncInfo&), ebpf::BPFModule::finalize_prog_func_info()::$_4>::_M_invoke(std::_Any_data const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, ebpf::FuncInfo&))
mold: error: undefined symbol: ebpf::ClangLoader::parse(std::unique_ptr<llvm::Module, std::default_delete<llvm::Module> >*, ebpf::TableStorage&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, char const**, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ebpf::ProgFuncInfo&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<int, std::tuple<int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<int>, std::allocator<std::pair<int const, std::tuple<int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >&)
>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::load_cfile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, char const**, int))>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::load_includes(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))>>> referenced by bpf_module.cc
>>> /usr/local/lib/libbcc.a(bpf_module.cc.o):(ebpf::BPFModule::load_c(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const**, int))>>> referenced 1 more times
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
Compiler version:
$ c++ --version
clang version 14.0.6 (https://github.com/llvm/llvm-project.git f28c006a5895fc0e329fe15fead81e37457cb1d1)
Is it possible that we can change Makefile to include api-static/clang-frontend in libbcc.a? I briefly looked at the CMakefile. Looks like they should be included in libbcc.a, but somehow they didn't. Could you do a investigation?
@yonghong-song it is entirely possible to do so. I'm no cmake expert, but in order to merge them all under the same library, it seems to me that instead of add_library, the sources should instead be added to the cmake list holding the source files for bcc-static.
I unfortunately can only afford limited time to spend on this, so I can't drive an investigation. I can, though, contribute to the discussion in this thread.
Based on how the cmake is laid out, the libraries are intently separate (i.e.: add_library intentionally creates a new lib for the C++ API and for the clang frontend). That would make sense to me if the original intent was to break down dependencies (e.g.: if you don't need the clang frontend, you shouldn't be required to link against clang, therefore you wouldn't link against libbcc-clang-frontend.a).
That said, I have absolutely no context on whether this was or not the original intent. Nor am I knowledgeable enough about the maintainer's community to know who to ask for such context.
@yonghong-song I failed to clarify this: if there's is a final decision that api-static and clang-frontend should be merged into bcc-static, I can go ahead and make the changes to the makefiles. I just can't afford to drive a consensus gathering for whether or not this is the way to go. So let me know which way this should be handled and I'll be happy to contribute the PR.
@juchem I think a single libbcc.a static library makes more sense. The library, esp. libbcc_clang_frontend, should be really invisible to users.
@yonghong-song the latest patch merges api-static and clang_frontend into libbcc. While working on this I realized the same problem happened to usdt-static and bcc-loader-static, so they got merged as well. libbcc should now be self-contained.
@yonghong-song I apologize for having deleted my fork by mistake. I got it restored. The latest version should reflect everything you asked for.
@yonghong-song I believe I have addressed all requests made so far. PR is rebased to latest master. Any chance you could give it another look? This affects everyone who links bcc statically and is very likely to be preventing them from upgrading bcc, as was my case, due to missing symbols.
@juchem https://github.com/iovisor/bcc/pull/4433 seems a simpler solution. Could you check whether that pull request works or not for you?