Linking errors on centos:7.9.2009 with Clang 13.0 and ROCm 4.3.1

[denisbertini]

I am facing a problem trying to link picongpu main program against the boost options and filesystem library. My settings are

• centos 7.9 (singularity container)
• Rocm 4.3.1 hipcc and Clang 13.0

H IP version: 4.3.21331-94fc2572
clang version 13.0.0 (https://github.com/RadeonOpenCompute/llvm-project roc-4.3.1 21313 286c48af238c2d3a24ebc5a06ea5191f333eaed0)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm-4.3.1/llvm/bin

• boost 1.75
• cmake 3.21

trying to link using such a system. i got the following problem:

[100%] Linking CXX executable picongpu
ld.lld: error: undefined symbol: boost::program_options::options_description::options_description(std::string const&, unsigned int, unsigned int)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(picongpu::SimulationStarter<picongpu::InitialiserController, picongpu::PluginController, picongpu::Simulation>::parseConfigs(int, char**))
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(picongpu::SimulationStarter<picongpu::InitialiserController, picongpu::PluginController, picongpu::Simulation>::parseConfigs(int, char**))
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(picongpu::SimulationStarter<picongpu::InitialiserController, picongpu::PluginController, picongpu::Simulation>::parseConfigs(int, char**))
>>> referenced 2 more times

ld.lld: error: undefined symbol: boost::program_options::arg
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(boost::program_options::typed_value<unsigned int, char>::name() const)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(boost::program_options::typed_value<unsigned short, char>::name() const)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(boost::program_options::typed_value<bool, char>::name() const)
>>> referenced 9 more times

ld.lld: error: undefined symbol: boost::program_options::invalid_option_value::invalid_option_value(std::string const&)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(void boost::program_options::validate<unsigned int, char>(boost::any&, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned int*, long))
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(void boost::program_options::validate<unsigned short, char>(boost::any&, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned short*, long))
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(void boost::program_options::validate<int, char>(boost::any&, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, int*, long))
>>> referenced 7 more times

ld.lld: error: undefined symbol: boost::program_options::validation_error::get_template(boost::program_options::validation_error::kind_t)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(boost::program_options::validation_error::validation_error(boost::program_options::validation_error::kind_t, std::string const&, std::string const&, int))

ld.lld: error: undefined symbol: boost::program_options::error_with_option_name::error_with_option_name(std::string const&, std::string const&, std::string const&, int)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(boost::program_options::validation_error::validation_error(boost::program_options::validation_error::kind_t, std::string const&, std::string const&, int))

ld.lld: error: undefined symbol: boost::program_options::validate(boost::any&, std::vector<std::string, std::allocator<std::string> > const&, bool*, int)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(boost::program_options::typed_value<bool, char>::xparse(boost::any&, std::vector<std::string, std::allocator<std::string> > const&) const)

ld.lld: error: undefined symbol: boost::program_options::validate(boost::any&, std::vector<std::string, std::allocator<std::string> > const&, std::string*, int)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(boost::program_options::typed_value<std::string, char>::xparse(boost::any&, std::vector<std::string, std::allocator<std::string> > const&) const)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(void boost::program_options::validate<std::string, char>(boost::any&, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::string, std::allocator<std::string> >*, int))

ld.lld: error: undefined symbol: boost::program_options::value_semantic_codecvt_helper<char>::parse(boost::any&, std::vector<std::string, std::allocator<std::string> > const&, bool) const
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(vtable for boost::program_options::typed_value<unsigned int, char>)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(vtable for boost::program_options::typed_value<unsigned short, char>)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(vtable for boost::program_options::typed_value<bool, char>)
>>> referenced 9 more times

ld.lld: error: undefined symbol: boost::program_options::error_with_option_name::substitute_placeholders(std::string const&) const
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(vtable for boost::wrapexcept<boost::program_options::validation_error>)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(vtable for boost::program_options::validation_error)
>>> referenced by main.cpp
>>>               CMakeFiles/picongpu.dir/main.cpp.o:(vtable for boost::wrapexcept<boost::program_options::invalid_option_value>)
>>> referenced 1 more times

ld.lld: error: undefined symbol: boost::program_options::abstract_variables_map::operator[](std::string const&) const
>>> referenced by ArgsParser.cpp
>>>               ArgsParser.cpp.o:(picongpu::ArgsParser::parse(int, char**)) in archive libpicongpu-hostonly.a

ld.lld: error: undefined symbol: boost::program_options::detail::cmdline::cmdline(std::vector<std::string, std::allocator<std::string> > const&)
>>> referenced by ArgsParser.cpp
>>>               ArgsParser.cpp.o:(boost::program_options::basic_command_line_parser<char>::basic_command_line_parser(int, char const* const*)) in archive libpicongpu-hostonly.a

ld.lld: error: undefined symbol: boost::program_options::detail::cmdline::set_additional_parser(boost::function1<std::pair<std::string, std::string>, std::string const&>)
>>> referenced by ArgsParser.cpp
>>>               ArgsParser.cpp.o:(boost::program_options::basic_command_line_parser<char>::extra_parser(boost::function1<std::pair<std::string, std::string>, std::string const&>)) in archive libpicongpu-hostonly.a

ld.lld: error: undefined symbol: boost::program_options::to_internal(std::string const&)
>>> referenced by ArgsParser.cpp
>>>               ArgsParser.cpp.o:(std::vector<std::string, std::allocator<std::string> > boost::program_options::to_internal<std::string>(std::vector<std::string, std::allocator<std::string> > const&)) in archive libpicongpu-hostonly.a

ld.lld: error: /lustre/rz/dbertini/gpu/softw/picongpu_softw/boost_1.75_0/lib/libboost_filesystem.so.1.74.0: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const@GLIBCXX_3.4.21 [--no-allow-shlib-undefined]
ld.lld: error: /lustre/rz/dbertini/gpu/softw/picongpu_softw/boost_1.75_0/lib/libboost_filesystem.so.1.74.0: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long)@GLIBCXX_3.4.21 [--no-allow-shlib-undefined]
ld.lld: error: /lustre/rz/dbertini/gpu/softw/picongpu_softw/boost_1.75_0/lib/libboost_filesystem.so.1.74.0: undefined reference to std::_V2::generic_category()@GLIBCXX_3.4.21 [--no-allow-shlib-undefined]
ld.lld: error: /lustre/rz/dbertini/gpu/softw/picongpu_softw/boost_1.75_0/lib/libboost_filesystem.so.1.74.0: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@GLIBCXX_3.4.21 [--no-allow-shlib-undefined]
ld.lld: error: /lustre/rz/dbertini/gpu/softw/picongpu_softw/boost_1.75_0/lib/libboost_filesystem.so.1.74.0: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()@GLIBCXX_3.4.21 [--no-allow-shlib-undefined]
ld.lld: error: /lustre/rz/dbertini/gpu/softw/picongpu_softw/boost_1.75_0/lib/libboost_filesystem.so.1.74.0: undefined reference to operator delete(void*, unsigned long)@CXXABI_1.3.9 [--no-allow-shlib-undefined]
ld.lld: error: /lustre/rz/dbertini/gpu/softw/picongpu_softw/boost_1.75_0/lib/libboost_filesystem.so.1.74.0: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_erase(unsigned long, unsigned long)@GLIBCXX_3.4.21 [--no-allow-shlib-undefined]
ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/picongpu.dir/build.make:136: picongpu] Error 1
make[1]: *** [CMakeFiles/Makefile2:166: CMakeFiles/picongpu.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

Any idea what could be wrong here ?

[psychocoderHPC]

Could you please post the command you used to build boost. It looks like you build boost for C++98 instead of C++11 or newer. The ABI for std::string in C++98 and C++11+ is not compatible therefore the symbol was not found.

Here are the commands how to build boost: https://picongpu.readthedocs.io/en/0.6.0/install/dependencies.html?highlight=boost#boost

If you build boost with the clang compiler you need to provide the information to boost too: ./bootstrap.sh --with-toolset=clang <all other params>

[denisbertini]

I used the following commands to build boost,

./bootstrap.sh --prefix=$SOFTW_PATH/boost_1.75_0 --with-libraries=atomic,chrono,context,date_time,fiber,filesystem,math,program_options,regex,serialization,system,thread
./b2 cxxflags="-std=c++14" -j 4 -q install

[denisbertini]

using the additional toolset option in bootstrap.sh i got

dbertini@lxbk0720:~/picongpu_softw/boost_1_75_0 > ./bootstrap.sh --with-toolset=clang --prefix=$SOFTW_PATH/boost_1.75_0 --with-libraries=atomic,chrono,context,date_time,fiber,filesystem,math,program_options,regex,serialization,system,thread 
Building B2 engine..

###
###
### Using 'gcc' toolset.
###
###

> /opt/rocm/bin/hipcc --version
HIP version: 4.3.21331-94fc2572
clang version 13.0.0 (https://github.com/RadeonOpenCompute/llvm-project roc-4.3.1 21313 286c48af238c2d3a24ebc5a06ea5191f333eaed0)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm-4.3.1/llvm/bin

###
###

> /opt/rocm/bin/hipcc -x c++ -std=c++11 -O2 -s -DNDEBUG builtins.cpp class.cpp command.cpp compile.cpp constants.cpp cwd.cpp debug.cpp debugger.cpp execcmd.cpp filesys.cpp frames.cpp function.cpp glob.cpp hash.cpp hcache.cpp hdrmacro.cpp headers.cpp jam.cpp jamgram.cpp lists.cpp make.cpp make1.cpp md5.cpp mem.cpp modules.cpp native.cpp object.cpp option.cpp output.cpp parse.cpp pathsys.cpp regexp.cpp rules.cpp scan.cpp search.cpp jam_strings.cpp startup.cpp subst.cpp sysinfo.cpp timestamp.cpp variable.cpp w32_getreg.cpp modules/order.cpp modules/path.cpp modules/property-set.cpp modules/regex.cpp modules/sequence.cpp modules/set.cpp execunix.cpp fileunix.cpp pathunix.cpp -o b2
> cp b2 bjam
tools/build/src/engine/b2
Unicode/ICU support for Boost.Regex?... not found.
Backing up existing B2 configuration in project-config.jam.1
Generating B2 configuration in project-config.jam for clang...

Bootstrapping is done. To build, run:

    ./b2
    
To generate header files, run:

    ./b2 headers

To adjust configuration, edit 'project-config.jam'.
Further information:

   - Command line help:
     ./b2 --help
     
   - Getting started guide: 
     http://www.boost.org/more/getting_started/unix-variants.html
     
   - B2 documentation:
     http://www.boost.org/build/

showing still using gcc toolset. May be one should ignore this ...

[denisbertini]

recompiling boost with the additional toolset option set to clang did not solve the linking issue

[denisbertini]

further investigations are showing that the problem comes from hipcc installed on our centos7.9 container. In the singularity def file we have

# On CentOS, install package centos-release-scl available in CentOS repository:
    yum install -y centos-release-scl

    # Install the devtoolset-7 collection:
    yum install -y devtoolset-7
    yum install -y devtoolset-7-libatomic-devel devtoolset-7-elfutils-libelf-devel

    # Install the ROCm rpms
    export ROCM_VERSION=4.3.1
    yum clean all
    echo -e "[ROCm]\nname=ROCm\nbaseurl=https://repo.radeon.com/rocm/yum/$ROCM_VERSION\nenabled=1\ngpgcheck=0" >> /etc/yum.repos.d/rocm.repo

    yum install -y rocm-dev rocm-validation-suite

this will i guess install Rocm package for the default centos7 gcc compiler which is not c++11 compatible, so the linking problem i guess .... I should then install a newer gcc compiler and manually install Rocm and compile the libraries for this compiler.

[psychocoderHPC]

Yes you should use a newer gcc.

The reason for

dbertini@lxbk0720:~/picongpu_softw/boost_1_75_0 > ./bootstrap.sh --with-toolset=clang --prefix=$SOFTW_PATH/boost_1.75_0 --with-libraries=atomic,chrono,context,date_time,fiber,filesystem,math,program_options,regex,serialization,system,thread 
Building B2 engine..

###
###
### Using 'gcc' toolset.
###
###

even if you select clang as toolset could be that you have no clang and clang++ in your PATH environment variable. You can check for both with which clang and which clang++ but maybe installing a newer gcc helps already.

[denisbertini]

Do you have any experience installing Rocm suite with spack?

[psychocoderHPC]

Do you have any experience installing Rocm suite with spack?

No sry, on your dev system I use apt and on other systems, I use whatever the admin is providing.

[BrianMarre]

@psychocoderHPC can this issue be closed?

[PrometheusPi]

This looks outdated. I will close it. Feel free to reopen if needed.