OpenFOAM escapes RPATH sanity check

[boegel]

Because the OpenFOAM binaries and libraries are in a non-standard location (i.e. outside of bin, lib, lib64), the RPATH sanity check is not effective for OpenFOAM (and other applications using non-standard paths).

This could be fixed by customising rpath_dirs in sanity_check_rpath in the OpenFOAM easyblock.

Once that is fixed, I expect the RPATH sanity check to fail when OpenFOAM is being installed under --rpath, because it uses CMake internally which by default strips RPATH specs (cfr. #1031).

cc @jdonners

[jdonners]

I see that, when built with an rpath, it now links to the dummy-version of libPstream.so (using EasyBuild 3.2.1):

[donners@int2 easybuild-easyblocks]$ LD_LIBRARY_PATH= ldd /hpc/eb/RedHatEnterpriseServer7/OpenFOAM/4.0-intel-2016b/OpenFOAM-4.0/platforms/linux64IccDPInt32Opt/bin/decomposePar|fgrep Pstr
	libPstream.so => /hpc/eb/RedHatEnterpriseServer7/OpenFOAM/4.0-intel-2016b/OpenFOAM-4.0/platforms/linux64IccDPInt32Opt/lib/dummy/libPstream.so (0x00002ae0d5c76000)

[bartoldeman]

I found out a while ago (July 2017 with OpenFOAM 4.1) that decomposePar is not linked using a direct -lPstream, but instead links to libfiniteVolume.so. This library in turn links to libPstream but also not via -lPstream but via -Xlinker --add-needed.

[boegel]

@besserox @wpoely86 @olesenm W.r.t. the linkage to lib/dummy/libPstream.so that @jdonners is seeing here when building OpenFOAM with the (experimental) EasyBuild RPATH support, that's bad right, it should be linking to lib/mpi/libPstream.so in order to get decent performance?

[olesenm]

The longer (but don't know if long enough) story: "running" with the dummy pstream is not a question of performance at all, but of linkage. It provides a dummy implementation of all the required library symbols, but if you try to use it for parallel communication it should complain and exit with an error message. In the normal case, you have provided a particular implementation of Pstream (eg, with openmpi, mpich, intelmpi backends) that is found first in the LD_LIBRARY_PATH.

Some utilities, with decomposePar being an example, are tagged internally as argList::noParallel() which means that although they have linkage to whatever Pstream flavour (openmpi etc, or even the dummy backend), parallel communication is never initialized by the utility. Any utilities that have been tagged as noParallel will be missing the -parallel option if you check their command line -help information.

[besserox]

A few comments:

• I believe it should be ok to link to lib/mpi/libPstream.so

• In the issue description:

Once that is fixed, I expect the RPATH sanity check to fail when OpenFOAM is being installed under --rpath, because it uses CMake internally which by default strips RPATH specs

AFAIK, OpenFOAM does not use CMake. It normally builds some third party software, and some of them use CMake. But when using EasyBuild, I don't think we build them.

• I think that OpenFOAM also uses dlopen() calls to load some libraries at runtime, like plugins. Because of that, a proper LD_LIBRARY_PATH should should be set anyway. But I think it is done by the source "${FOAM_BASH}". So it should not be a problem.

[boegel]

@jdonners Are the other utilities (which do have the -parallel option) also linking to lib/dummy/libPstream.so? Is lib/mpi/libPstream.so actually there?

[olesenm]

Not really sure what is going wrong here. If take a look at the last few lines of etc/config.sh/mpi there is a line prepending ... /lib/$FOAM_MPI so this should be found first (before possibly hitting the dummy fallback). Further checking the build bits for src/Pstream the libmpi.so should be build into this directory using the WM_MPLIB rules.

However, in some cases in the past, we observed situations where the "touch" for the sentinel file failed for inexplicable reasons. This should be fixed, but maybe something like that has come back to haunt us?