Build failed on LLNL singe cluster

[Plasdom]

I'm trying to build the latest version of bout++ on LLNL's singe cluster (Red Hat Enterprise Linux v7.9). I am using the singe modules gcc (v12.2) and netcdf (v4.3.3.1), and versions of cmake (v3.30.1), mpich (v4.2.2) and fftw (v3.3) built from source. I can succesfully configure but not build with the following:

module load gcc/12.2
module load netcdf
export PATH=/home/power8/installs/cmake/bin:$PATH
export PATH=/home/power8/installs/mpich-4.2.2/bin:$PATH
export PATH=/home/power8/installs/fftw-3.3/bin:$PATH
cmake BOUT-dev -B build-bout-default

The error from cmake is many lines similar to:

/usr/bin/ld: /home/power8/installs/fftw-3.3/lib/libfftw3.a(apiplan.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC

I've also tried running cmake with the flag -DBUILD_SHARED_LIBS=Off. This allows bout++ to build successfully, but it then fails when building examples:

cmake --build build-bout-default --target conduction
/usr/bin/ld: Dwarf Error: found dwarf version '5', this reader only handles version 2, 3 and 4 information.
../../lib/libbout++.a(options_netcdf.cxx.o): In function `(anonymous namespace)::findDimension(netCDF::NcGroup&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int)':
options_netcdf.cxx:(.text+0xc0): undefined reference to `netCDF::NcGroup::getDim(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcGroup::Location) const'
options_netcdf.cxx:(.text+0xdb): undefined reference to `netCDF::NcGroup::addDim(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long) const'
/usr/bin/ld: Dwarf Error: found dwarf version '12', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x126): undefined reference to `netCDF::NcGroup::getDim(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcGroup::Location) const'
/usr/bin/ld: Dwarf Error: found dwarf version '4100', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x165): undefined reference to `netCDF::NcGroup::addDim(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long) const'
/usr/bin/ld: Dwarf Error: found dwarf version '491', this reader only handles version 2, 3 and 4 information.
../../lib/libbout++.a(options_netcdf.cxx.o): In function `(anonymous namespace)::getCurrentTimeIndex(netCDF::NcVar const&)':
options_netcdf.cxx:(.text+0xc54): undefined reference to `netCDF::NcVar::getAtts[abi:cxx11]() const'
/usr/bin/ld: Dwarf Error: found dwarf version '133', this reader only handles version 2, 3 and 4 information.
../../lib/libbout++.a(options_netcdf.cxx.o): In function `(anonymous namespace)::writeGroup(Options const&, netCDF::NcGroup, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
options_netcdf.cxx:(.text+0x1c01): undefined reference to `netCDF::NcGroup::getVar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcGroup::Location) const'
options_netcdf.cxx:(.text+0x1c4d): undefined reference to `netCDF::NcGroup::addVar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcType const&, std::vector<netCDF::NcDim, std::allocator<netCDF::NcDim> > const&) const'
/usr/bin/ld: Dwarf Error: found dwarf version '2049', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x1e58): undefined reference to `netCDF::NcGroup::getGroup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcGroup::GroupLocation) const'
/usr/bin/ld: Dwarf Error: found dwarf version '1763', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x1e72): undefined reference to `netCDF::NcGroup::addGroup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
/usr/bin/ld: Dwarf Error: found dwarf version '18689', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x1fbe): undefined reference to `netCDF::NcGroup::getDim(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcGroup::Location) const'
/usr/bin/ld: Dwarf Error: found dwarf version '14336', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x1ff0): undefined reference to `netCDF::NcGroup::addDim(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
/usr/bin/ld: Dwarf Error: found dwarf version '6285', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x2af3): undefined reference to `netCDF::NcVar::putAtt(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcType const&, int) const'
/usr/bin/ld: Dwarf Error: found dwarf version '426', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x2c31): undefined reference to `netCDF::NcVar::putAtt(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcType const&, int) const'
/usr/bin/ld: Dwarf Error: found dwarf version '9', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x2c6b): undefined reference to `netCDF::NcVar::putAtt(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
/usr/bin/ld: Dwarf Error: found dwarf version '47371', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x2cb6): undefined reference to `netCDF::NcVar::putAtt(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcType const&, double) const'
/usr/bin/ld: Dwarf Error: found dwarf version '32512', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x2dda): undefined reference to `netCDF::NcVar::putAtt(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcType const&, int) const'
/usr/bin/ld: Dwarf Error: found dwarf version '5', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x2eb7): undefined reference to `netCDF::NcVar::putAtt(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcType const&, int) const'
/usr/bin/ld: Dwarf Error: found dwarf version '38160', this reader only handles version 2, 3 and 4 information.
../../lib/libbout++.a(options_netcdf.cxx.o): In function `bout::OptionsNetCDF::write(Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
options_netcdf.cxx:(.text+0x34e7): undefined reference to `netCDF::NcFile::NcFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcFile::FileMode)'
/usr/bin/ld: Dwarf Error: found dwarf version '512', this reader only handles version 2, 3 and 4 information.
../../lib/libbout++.a(options_netcdf.cxx.o): In function `(anonymous namespace)::verifyTimesteps(netCDF::NcGroup const&)':
options_netcdf.cxx:(.text+0x37fe): undefined reference to `netCDF::NcGroup::getVars[abi:cxx11](netCDF::NcGroup::Location) const'
/usr/bin/ld: Dwarf Error: found dwarf version '50006', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x3840): undefined reference to `netCDF::NcVar::getAtts[abi:cxx11]() const'
/usr/bin/ld: Dwarf Error: found dwarf version '3384', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x38d7): undefined reference to `netCDF::NcAtt::getValues(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) const'
options_netcdf.cxx:(.text+0x38f0): undefined reference to `netCDF::NcGroup::getDim(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcGroup::Location) const'
/usr/bin/ld: Dwarf Error: found dwarf version '1836', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x3aff): undefined reference to `netCDF::NcGroup::getGroups[abi:cxx11](netCDF::NcGroup::GroupLocation) const'
/usr/bin/ld: Dwarf Error: found dwarf version '4111', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x3c8c): undefined reference to `netCDF::NcDim::getName[abi:cxx11]() const'
../../lib/libbout++.a(options_netcdf.cxx.o): In function `bout::OptionsNetCDF::verifyTimesteps() const':
options_netcdf.cxx:(.text+0x46ab): undefined reference to `netCDF::NcFile::NcFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcFile::FileMode)'
/usr/bin/ld: Dwarf Error: found dwarf version '31894', this reader only handles version 2, 3 and 4 information.
../../lib/libbout++.a(options_netcdf.cxx.o): In function `(anonymous namespace)::readGroup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcGroup const&, Options&)':
options_netcdf.cxx:(.text+0x484d): undefined reference to `netCDF::NcGroup::getVars[abi:cxx11](netCDF::NcGroup::Location) const'
/usr/bin/ld: Dwarf Error: found dwarf version '4152', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x4e0a): undefined reference to `netCDF::NcVar::getAtts[abi:cxx11]() const'
/usr/bin/ld: Dwarf Error: found dwarf version '24', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x505f): undefined reference to `netCDF::NcGroup::getGroups[abi:cxx11](netCDF::NcGroup::GroupLocation) const'
/usr/bin/ld: Dwarf Error: found dwarf version '57215', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text+0x582f): undefined reference to `netCDF::NcAtt::getValues(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) const'
/usr/bin/ld: Dwarf Error: found dwarf version '14336', this reader only handles version 2, 3 and 4 information.
../../lib/libbout++.a(options_netcdf.cxx.o): In function `bout::OptionsNetCDF::read()':
options_netcdf.cxx:(.text+0x6c0e): undefined reference to `netCDF::NcFile::NcFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcFile::FileMode)'
/usr/bin/ld: Dwarf Error: found dwarf version '6339', this reader only handles version 2, 3 and 4 information.
../../lib/libbout++.a(options_netcdf.cxx.o): In function `(anonymous namespace)::writeGroup(Options const&, netCDF::NcGroup, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [clone .cold]':
options_netcdf.cxx:(.text.unlikely+0x8f6): undefined reference to `netCDF::NcType::getName[abi:cxx11]() const'
/usr/bin/ld: Dwarf Error: found dwarf version '32512', this reader only handles version 2, 3 and 4 information.
options_netcdf.cxx:(.text.unlikely+0x921): undefined reference to `netCDF::NcType::getName[abi:cxx11]() const'
collect2: error: ld returned 1 exit status
gmake[3]: *** [examples/conduction/conduction] Error 1
gmake[2]: *** [examples/conduction/CMakeFiles/conduction.dir/all] Error 2
gmake[1]: *** [examples/conduction/CMakeFiles/conduction.dir/rule] Error 2

The cache file is here: CMakeCache.txt

[bendudson]

Hey @Plasdom ! For the first error: It looks like you have to reinstall fftw3 as a shared library. I think the second (DWARF) error is because the system binutils /usr/bin/ld is probably very old and can't parse the object files (I'm not sure if it's a BOUT++ or netcdf object).

[Plasdom]

Thanks for the help. I've rebuilt fftw with --enable-shared and removed the flag -DBUILD_SHARED_LIBS=Off from the bout++ build command, and can build successfully. The tests are failing due to not being able to find netcdf functions:

/usr/bin/ld: warning: libnetcdf.so.7, needed by /usr/local/netcdf/lib/libnetcdf_c++4.so, may conflict with libnetcdf.so.15
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcFile::NcFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcFile::FileMode)'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcGroup::addVar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcType const&, std::vector<netCDF::NcDim, std::allocator<netCDF::NcDim> > const&) const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcGroup::getVars[abi:cxx11](netCDF::NcGroup::Location) const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcVar::putAtt(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcType const&, double) const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcType::getName[abi:cxx11]() const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcGroup::addGroup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcVar::putAtt(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcType const&, int) const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcGroup::getGroup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcGroup::GroupLocation) const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcGroup::getGroups[abi:cxx11](netCDF::NcGroup::GroupLocation) const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcGroup::getDim(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcGroup::Location) const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcAtt::getValues(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcVar::getAtts[abi:cxx11]() const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcGroup::addDim(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcVar::putAtt(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcGroup::addDim(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long) const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcDim::getName[abi:cxx11]() const'
../../lib/libbout++.so.5.1.0: undefined reference to `netCDF::NcGroup::getVar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcGroup::Location) const'
collect2: error: ld returned 1 exit status
gmake[3]: *** [examples/conduction/conduction] Error 1
gmake[2]: *** [examples/conduction/CMakeFiles/conduction.dir/all] Error 2
gmake[1]: *** [examples/conduction/CMakeFiles/conduction.dir/rule] Error 2
gmake: *** [conduction] Error 2

The warning on the first line is probably a clue. The version of netcdf I'm using is 4.7.0. Looks like the version of binutils is 2.27 which is quite old, so do I need to update that?

[Plasdom]

Update: I have built the latest version of binutils and get the same error. The warning on the first line above was caused by using a different version of gcc from the one it seems netcdf was compiled with; fixing this removes the warning but does not fix the errors (undefined reference to ...)

[dschwoerer]

Can you post the content of examples/conduction/CMakeFiles/conduction.dir/link.txt?

Have you tried removing the build folder, after you fixed the netcdf warning?

[Plasdom]

Yes, examples/conduction/CMakeFiles/conduction.dir/link.txt looks like:

This is after removing the build folder and rebuilding.

[dschwoerer]

On 8/5/24 15:32, Dominic Power wrote:

Yes, |examples/conduction/CMakeFiles/conduction.dir/link.txt| looks like: image.png (view on web) https://github.com/user-attachments/assets/f4f73baf-f171-4ccb-8fe7-503bd0e36d36

Can you please share the text rather then a picture?

[Plasdom]

/usr/local/gcc-7.4.0/bin/g++ -O2 -g -DNDEBUG -Wl,-rpath -Wl,/home/power8/installs/mpich-4.2.2/lib -Wl,--enable-new-dtags CMakeFiles/conduction.dir/conduction.cxx.o -o conduction  -Wl,-rpath,/home/power8/bout/build-bout/lib:/home/power8/installs/mpich-4.2.2/lib:/usr/local/netcdf/lib:/home/power8/installs/fftw/lib ../../lib/libbout++.so.5.1.0 ../../lib/libfmt.so.10.1.1 ../../lib/libpvode.so.1.0.0 ../../lib/libpvpre.so.1.0.0 /home/power8/installs/mpich-4.2.2/lib/libmpicxx.so /home/power8/installs/mpich-4.2.2/lib/libmpi.so /usr/local/netcdf/lib/libnetcdf_c++4.so /usr/lib64/libnetcdf.so /home/power8/installs/fftw/lib/libfftw3.so /usr/lib64/libuuid.so -ldl

[dschwoerer]

what does

nm -CD /usr/local/netcdf/lib/libnetcdf_c++4.so | grep 'netCDF::NcFile::NcFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, netCDF::NcFile::FileMode)'

say?

Is /usr/local/netcdf/lib/libnetcdf_c++4.so really the netcdf-c++ library you want to use? /usr/lib64/libnetcdf.so also looks wrong, as you mentioned you have loaded a netcdf module.

You probably need to tell cmake to use the ones you want to use.

[dschwoerer]

if the above has no output, what does: nm -CD /usr/local/netcdf/lib/libnetcdf_c++4.so | grep 'netCDF::NcFile::NcFile' give?

[Plasdom]

The first command has no output. The second has the following:

0000000000026ef0 T netCDF::NcFile::NcFile(std::string const&, netCDF::NcFile::FileMode)
0000000000027080 T netCDF::NcFile::NcFile(std::string const&, netCDF::NcFile::FileMode, netCDF::NcFile::FileFormat)
0000000000026dc0 T netCDF::NcFile::NcFile()
0000000000026ef0 T netCDF::NcFile::NcFile(std::string const&, netCDF::NcFile::FileMode)
0000000000027080 T netCDF::NcFile::NcFile(std::string const&, netCDF::NcFile::FileMode, netCDF::NcFile::FileFormat)
0000000000026dc0 T netCDF::NcFile::NcFile()

[dschwoerer]

Is /usr/local/netcdf/lib/libnetcdf_c++4.so really the netcdf-c++ library you want to use? /usr/lib64/libnetcdf.so also looks wrong, as you mentioned you have loaded a netcdf module.

Any thoughts on this?

[dschwoerer]

As far as I can tell, the issue is that netcdf-c++ has been compiled against an old c++-standard, while you are using a newer c++ standard to compile bout++. As such you need to link against a netcdf-c++ that is also compiled with a newer c++ standard.

See previous comments as to what could be the cause of that.

[Plasdom]

I thought it might be something like that. I've tried building netcdf from source with no success so far, so I'll focus on that for now. Thanks for your help.

[ZedThree]

Have you tried building BOUT++ with:

cmake . -B build -DBOUT_DOWNLOAD_NETCDF_CXX4=ON

This will build the netcdf C++ library as part of the BOUT++ build.

It also looks like you're using the system gcc (7.4) rather than the modules one (12.2). I suspect you're also picking up the system netcdf too. Please can you run the following:

g++ --version
nc-config --all
mpirun --version
mpic++ -show

It could be that you've built mpich using the system compiler, so cmake is going to pick that up instead of the module one. If so, you'll either have to recompile mpich or use an MPI implementation from the module system

[Plasdom]

Have you tried building BOUT++ with: cmake . -B build -DBOUT_DOWNLOAD_NETCDF_CXX4=ON

Yep, as you spotted BOUT++ is picking up the system netcdf and proceeding without building it, even with that flag set. I suspect building netcdf myself and using that with BOUT++ would fix the problem, but I'm hitting an error there which I haven't resolved yet (ddispatch.c:28:10: fatal error: /usr/local/include/curl/curl.h: Permission denied).

Output from g++ --version:

g++ (GCC) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Output from nc-config --all:

This netCDF 4.3.3.1 has been built with the following features: 

  --cc        -> gcc
  --cflags    ->  -I/usr/include -I/usr/include/hdf
  --libs      -> 

  --has-c++   -> no
  --cxx       -> 
  --has-c++4  -> yes
  --cxx4      -> g++

  --fc        -> gfortran
  --fflags    -> -I/usr/include
  --flibs     -> -lnetcdff  
  --has-f90   -> yes

  --has-dap   -> yes
  --has-nc2   -> yes
  --has-nc4   -> yes
  --has-hdf5  -> yes
  --has-hdf4  -> yes
  --has-pnetcdf-> no

  --prefix    -> /usr
  --includedir-> /usr/include
  --version   -> netCDF 4.3.3.1

Output from mpirun --version:

HYDRA build details:
    Version:                                 4.2.2
    Release Date:                            Wed Jul  3 09:16:22 AM CDT 2024
    CC:                              /usr/local/gcc-12.2/bin/gcc      
    Configure options:                       '--disable-option-checking' '--prefix=/home/power8/installs/mpich-4.2.2' '--with-hwloc=embedded' 'CC=/usr/local/gcc-12.2/bin/gcc' 'CXX=/usr/local/gcc-12.2/bin/g++' 'FC=/usr/local/gcc-12.2/bin/gfortran' '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -DNETMOD_INLINE=__netmod_inline_ofi__ -I/home/power8/mpich-4.2.2/src/mpl/include -I/home/power8/mpich-4.2.2/modules/json-c -I/home/power8/mpich-4.2.2/modules/hwloc/include -D_REENTRANT -I/home/power8/mpich-4.2.2/src/mpi/romio/include -I/home/power8/mpich-4.2.2/src/pmi/include -I/home/power8/mpich-4.2.2/modules/yaksa/src/frontend/include -I/home/power8/mpich-4.2.2/modules/libfabric/include'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Demux engines available:                 poll select

Output from mpic++ -show:

/usr/local/gcc-12.2/bin/g++ -I/home/power8/installs/mpich-4.2.2/include -L/home/power8/installs/mpich-4.2.2/lib -lmpicxx -Wl,-rpath -Wl,/home/power8/installs/mpich-4.2.2/lib -Wl,--enable-new-dtags -lmpi

[dschwoerer]

CMake is sometimes weird about finding the wrong compiler. I found I often need something like:

cmake -DCMAKE_CXX_COMPILER=$(which g++) \
    -DCMAKE_C_COMPILER=$(which gcc) \
    -DMPI_CXX_COMPILER=$(which mpic++) \
    -DMPI_C_COMPILER=$(which mpicc)

maybe something like that helps cmake find the right compiler + mpiwrappers

Setting the serial compiler is sometimes needed, as otherwise cmake things the flags do not work, because it tests them with the wrong compiler.

[ggeorgakoudis]

@Plasdom I want to point out that we also have a repo for BOUT++ environment configurations at https://github.com/boutproject/BOUT-configs. It uses spack to create the environment, which should avoid the burden to manually collect and build dependencies and compiler incompatibilities from pre-built binaries.

Although we don't have a configuration for singe, we have one for perlmutter which you might be able to use as a blueprint (there is also one for lassen, but it is deprecated and will be updated soon-ish).